CTSA OMB stmt B 091710

Supporting Statement B for

National Evaluation of the
Clinical and Translational Science Awards (CTSA) Initiative (NCRR)

September 17, 2010

Patricia Newman

Program Analyst

Office of Science Policy

National Center for Research Resources

6701 Democracy Boulevard

MSC 4874

Bethesda, Maryland 20892-4874.

Telephone: 301-435-0864

Fax: 301-480-3654

Email: [email protected]

Table of contents

Supporting Statement B. Collections of Information Employing Statistical Methods 3

B.1. Respondent Universe and Sampling Methods 3

B.2. Procedures for the Collection of Information 3

B.2.1. Statistical Methodology for Stratification and Sample Selection 3

B.2.2. Estimation Procedure 4

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Jurisdiction 5

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures 5

B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles 5

B.3. Methods to Maximize Response Rates and Deal With Non-response 6

B.4. Tests of Procedures or Methods To Be Undertaken 6

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 7

Exhibit B1. Estimated size of universe to be sampled 3

Supporting Statement B. Collections of Information Employing Statistical Methods

This data collection will include sampling of users and nonusers for the utilization study. For the trainees and scholar survey, a census of those funded by the CTSA education and training awards will be included. For the mentors survey, all mentors reported in the past two years of Non-Competing Continuation Progress Reports (PHS 2590) (2009 and 2010) will be surveyed. Since there is a time lag between the submission of the PHS 2590 and the initiation of the surveys, updates will be requested from local CTSA contacts to ensure sampling accuracy.

B.1. Respondent Universe and Sampling Methods

The universe of potential respondents to be included in this data collection is shown in Exhibit 5. Since the full universe of trainees/scholars and those who have mentored in the last two years will be included, no statistical sampling methods will be implemented for those surveys. Sampling methods for the users and nonusers surveys are detailed below.

Exhibit B1. Estimated size of universe to be sampled

Population	Universe size	Sample size
Users	18,428	700
Nonusers	68,690	2,000
Trainees/scholars	1,516	1,516
Mentors	1,688	1,688
Total	90,322	5,904

B.2. Procedures for the Collection of Information

B.2.1. Statistical Methodology for Stratification and Sample Selection

We intend to get 500 completes for both the users survey and the nonusers survey in order to have data that provide a reliable picture of usage. Given that target, we will draw a stratified random sample of 700 users and 2,000 nonusers, stratified by awardee cohort. Based on experience with similar surveys and consultation with CTSA consortium evaluation experts, we expect the response rate for nonusers to be lower than that for users. We will use this initial sampling to take a close look at the percentage of nonusers who complete the survey and indicate that they have not heard of the CTSA Initiative or the CTSA at their institution. If too few nonusers report knowledge of the CTSA, we may need to draw a supplemental sample. Statistical experts determined that these sample sizes are large enough to detect statistical differences, if they exist, based on past experience with similar surveys.

Surveys for CTSA trainees/scholars and mentors will be sent to the universe of potential respondents. These are defined (respectively) as trainees/scholars currently receiving CTSA education or training funding or having completed a CTSA-funded education/training program, and the mentors who have been reported to have been formally affiliated with such a program in the last two years. These will be drawn from the 46 awardee institutions in the first four CTSA awardee cohorts.

B.2.2. Estimation Procedure

Each of the four surveys will be analyzed using descriptive and inferential statistical procedures. First, item-wise descriptive statistics for each survey will be calculated. In the case of users and nonusers surveys, initial descriptive statistics will provide a detailed snapshot of how scientists and researchers are using (or not using) CTSA-provided resources, which type of resources they anticipate needing in the future, and perceptions of the benefits and problems associated with using these resources. Subsequent multivariate analysis will use General Linear Models (GLM) and Hierarchical Linear Models (HLM) to identify whether scientists’ needs, usage, experiences, and associated outcomes differ across the key phases of translational research, across key demographic or professional groups, or across CTSA-affiliated institutions with different institutional/organizational characteristics.

In the case of the trainees/scholars surveys, the item-wise descriptive statistics will be used to identify typical characteristics of CTSA-supported training programs and program participants; this includes experiences and attitudes toward key phases of translational research, the perceived adequacy of CTSA-supported training and mentoring, career intentions, and participation in career development activities. GLM and HLM will be used to identify the extent to which participant-reported outcomes and research-related attitudes and intentions vary across research interests and expertise, across demographic groups, and by experiences in the CTSA-supported program (e.g., training activities and mentoring experiences).

Item-wise descriptive statistics for the mentor surveys will be used to identify the characteristics of formal mentors in CTSA-supported programs and their experiences with the CTSA Initiative and with their mentees. GLM and HLM will be used to identify how key mentor characteristics and background relate to mentoring expectations, mentor-mentee interactions, and experiences with the mentoring program; how these factors relate to perceptions of the value of the mentoring program; and how these variables and their relationships differ across CTSA-supported programs.

Next, each survey will be analyzed using multivariate statistical techniques, including GLM and HLM. Where appropriate, HLM will be used to take into account the variance in certain types of regression relationships across different CTSA-affiliated institutions. This will provide a more accurate estimate of the statistical significance and standard error associated with a particular regression relationship, more accurate estimates of effects for CTSA-affiliated institutions with smaller numbers of respondents, as well as additional information on the extent to which such relationships vary (or are constant) across the institutions. Results of inferential analyses will be reported in terms of statistical significance tests (e.g., p-values, the extent to which a given result is likely to have resulted by chance) and effect sizes (e.g., regression weights, providing an estimate of the magnitude of a given relationship).

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Jurisdiction

To meet the purposes described in the justification, statistical analyses will be interpreted in terms of both effect sizes and statistical significance. In terms of statistical significance, a baseline alpha criterion of 0.05 will be used to identify significant effects.

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

No unusual problems are anticipated.

B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles

As described previously, the proposed data collection will lay the foundation for data to be collected over time. It is expected that the surveys will be administered on a biennial basis.

B.3. Methods to Maximize Response Rates and Deal With Non-response

Before the surveys are sent, NCRR will send a letter to the CTSA PIs to inform them of the effort, encourage their cooperation, and ask for a local contact (e.g., program administrator) to be named. Westat will then provide the local contact with additional details about the surveys and collect information necessary to maximize response rates on behalf of NCRR.

Users and mentors surveys will be emailed to the most recent email address available in the NIH data files taken from Non-Competing Continuation Progress Reports (PHS 2590) submitted by awardee institutions. Trainees/scholars surveys will be emailed to the most recent email address available in these files, supplemented by updated contact information for education/training funding recipients who have left the institution, as provided by the local CTSA contact. Nonusers’ surveys will be emailed to the most recent email address available in the NIH data files that include contact information for recipients of all NIH awards.

Should any of the emails be automatically returned due to problems with the email address, Westat project staff will conduct an Internet search to try to locate a more current email address for the individual. We expect that barring extremely unusual circumstances, we will be able obtain up-to-date contact information for the majority of potential respondents selected to participate. In the cover email that accompanies the survey link, respondents will be asked to complete the survey online by a specified date approximately two calendar weeks from the date of receipt. The letter will also contain the names, email addresses, and telephone numbers of Westat staff available to assist respondents with any questions or concerns they may have about the survey or the evaluation. Immediately after the due date for the survey, Westat will contact those who have not yet returned their surveys to remind them that the surveys are now past due. Respondents will receive up to two emails and then up to three follow-up telephone calls at one-week intervals. As described in Section A, we expect approximately 71 percent of users, 25 percent of nonusers, and 80 percent of both trainees/scholars and mentors to respond.

B.4. Tests of Procedures or Methods To Be Undertaken

All four data collection instruments have been reviewed by experts in the field to ensure that the methods will be properly implemented. Experts within NCRR, other NIH institutes and centers, and evaluators associated with the CTSA consortium have been consulted about the overall methodology as well as the specific data collection methods to be used in this evaluation. In addition, as described in Supporting Statement A, Section A.8, 23 informants from CTSA awardees (seven users, five nonusers, seven trainees/scholars, four mentors) and two experts in cognitive testing have been consulted in development and testing of the surveys.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following individuals were consulted on the study design.

Agency Unit

Reneé Joskow, Program Officer, Division of Clinical Research Resources, National Center for Research Resources, National Institutes of Health, 301-435-0961

Carol Merchant, Medical Officer, Division for Clinical Research Resources, National Center for Research Resources, National Institutes of Health, 301-480-3661

Lori Mulligan, Director, Office of Science Policy, National Center for Research Resources, National Institutes of Health, 301-435-0897

Patricia Newman, Program Analyst, Office of Science Policy, National Center for Research Resources, National Institutes of Health, 301-435-0864

Iris Obrams, Deputy Director, Division for Clinical Research Resources, National Center for Research Resources, National Institutes of Health, 301-435-0768.

Lawrence Solomon, Senior Health Science Analyst, Office of Science Planning and Assessment, National Cancer Institute, National Institutes of Health, 301-451-9985.

Meryl Sufian, Supervisory Health Science Policy Analyst, Office of Science Policy, National Center for Research Resources, National Institutes of Health, 301-435-0757

CTSA Consortium

Susan Pusek, Director, Education Programs, North Carolina Translational and Clinical
Sciences (TRaCS) Institute, Gillings School of Public Health, University of North Carolina, 919-966-0128

Doris Rubio, Associate Professor of Medicine, Biostatistics, Nursing, and Clinical and Translational Science, Division of General Internal Medicine, University of Pittsburgh,
412-692-2023

Contractor

Kerry Levin, Associate Director, Westat (Cognitive Testing), 301-738-3563

Cynthia Robins, Senior Study Director, Westat (Cognitive Testing), 301-738-3524

Westat will be responsible for data collection and analysis under the direction of Dr. Joy Frechtling
301-517-4006 or [email protected]. The following individuals will conduct data collection and analysis.

Edward Mann, Senior Systems Analyst, 301-294-4434

Randy Herbison, Senior Systems Analyst, 240-453-2661

Andrew Slaughter, Research Analyst, 301-212-2154