Supporting Statement B For:
Cancer Trials Support Unit (CTSU)
Public Use Forms and Customer Satisfaction Surveys (NCI)
May 21, 2010
Michael Montello, Pharm. D.
CTEP
National Cancer Institute
6130 Executive Blvd.
Rockville, MD 20852
Phone: (301) 435-9206
Fax: (301) 402-0557
Table of Contents
B. STATISTICAL METHODS 1
B.1 Respondent Universe and Sampling Methods 1
B.2 Procedures for the Collection of Information 2
B.3 Methods to Maximize Response Rates and Deal with Nonresponse 4
B.4 Test of Procedures or Methods to be Undertaken 4
B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 5
List of Attachments referenced in SSB
Attachment 2 – Web Site Survey
Attachment 3 – Help Desk Survey
Attachment 4 – Oncology Patient Enrollment Network (OPEN) Survey
Attachment 6 - Westat IRB Letter and OHSR Determination
The CTSU collects three satisfaction surveys to objectively measure customer satisfaction of select CTSU applications and services. Simple descriptive statistics are used for analysis of data.
The surveys target users of CTSU system/services within the recent past. Approximately 705 persons are randomly sampled for the web site and help desk surveys from a larger pool of potential respondents in 2009. For the most recent help desk survey, 1,250 (55%) e-mail addresses were randomly selected from the population of 2,274 CTSU users who had contacted the CTSU Help Desk in January and February of 2009. Of the 1,250 users, there were 409 respondents. This represents a 33% response rate for the survey or a sample of approximately 18% of the currently active user community. No other filtering or selection criteria were applied to selection of the names or e-mail addresses.
For the most recent web site survey, 985 names (15%) were randomly selected from the population of approximately 6,441 registered CTSU users who had visited the CTSU ‘members’ web site (http://members.ctsu.org) within the last 12 months. The 12-month period was used to filter out users who may have invalid e-mail addresses or no current experience with the web site. Still some of these users represented may be inactive, since there is no way to verify the current CTSU participation status. Of the 985 names, there were 296 respondents. This represents a 30% response rate for the survey or a sample of approximately 5% of the currently active user community. No other filtering or selection criteria were applied to selection of the names or e-mail addresses.
For the Oncology Patient Enrollment Network (OPEN) survey, all users of the OPEN application are eligible to complete a survey. There is no consideration of other characteristics beyond use of the services/applications. In early 2010 there was a respondent pool of 600 OPEN application users.
Survey Procedures.
The project team conducts a formal meeting to develop the survey questions. Survey questions fall into three general categories, frequency of use, general satisfaction, and specific questions on a new services or applications. The surveys are made up of 15 to 20 multiple choice questions with a free text field for additional comments at the end of the survey. The help desk and web site surveys have 5 to 8 core questions used quantitatively demonstrate trends in user satisfaction over time. (See attachments 2 and 3) Other questions are used to measure user satisfaction with new features or services, and to determine the frequency of application/service use by the respondent. CTSU will submit a change request to the OMB packet if survey questions are added or altered. Questions are kept to a minimum to decrease the time necessary to complete the survey and encourage response.
The OPEN survey is an ongoing survey available to persons completing an enrollment in the OPEN application. (See attachment 4) A set of core questions was developed, but these may vary over time as new features are added to the system.
The survey populations for the help desk and web site surveys are randomly selected from a pool of approximately 1000 recent users of the help desk or web site over the previous 6 months. (Response rates are generally <30% and response rates are reflected in table A-12-1 in the SSA.) To avoid bias there is no attempt to stratify users. Several methods are used to obtain objective results:
To avoid sampling bias, surveys questionnaires are sent broadly to the customer base or randomly selected samples of the customers.
Identification of individual respondents is withheld from the staff involved in the service being surveyed.
To avoid over interpretation of single survey results, trends are measured over time.
Whenever possible, a core set of satisfaction measures are used over time.
Customers are provided opportunity for open-ended responses.
Survey notifications are sent by e-mail that includes instructions and a link. The survey is deployed using Zoomerang™, an online survey system. Reminder e-mails are sent to participants that have not yet responded encouraging participation approximately a week after the initial e-mail. The survey is closed 10 business days after distribution.
All information collected is related to user experience with CTSU services and applications. No identifying information is collected in the survey and no questions of a sensitive nature are asked in the survey. E-mail information is not used within the analysis.
Following survey deployment, the project team reviews the survey data to identify recommendations for continuous improvement activities. Analysis tools such as fishbone diagrams, flow charts, and/or SWOT (Strengths, Weaknesses, Opportunities, and Threats) are used to compare data relationships to products and services. This process will include identifying:
A summary of strengths identified by the customer;
A summary of weaknesses and customer concerns;
Recommendations for leveraging strengths identified by the survey data; and
Opportunities for improvement and priority recommendations.
Survey results for all multiple choice questions are categorized as “satisfied”, “neutral” and “dissatisfied.” These categories are selected based upon a Likert scale with 1 and 2 response categories as “dissatisfied”, a response of 3 is “neutral,” and responses of 4 or 5 are categorized as “satisfied.” Results are compared across surveys to show general customer satisfaction trends.
In addition, free text comments are reviewed and categorized by area. Often comments are related to other CTSU processes, and these are shared with the team leaders from the appropriate task area.
The project team drafts the survey findings report using a standard survey template. Survey results are reported to CTEP using the Survey Findings Template that describes the following areas:
Survey Objectives
Methodology
Results (summary, open ended questions, core question)
Important findings
Conclusions
Recommendations
Rationale for Sample Size.
For the help desk and web site surveys, a sample of approximately 1,000 participants was selected to avoid bias and achieve a significant number of respondents. Response rates are generally less than 30% and are reflected in table A.12-1 in the SSA.
Quality Control.
The contractor will follow a standard operating procedure developed for the CTSU surveys. The CTSU Standard Operating Procedures are reviewed every 6 months, and staff is trained on procedures in their work area.
The CTSU Project was reviewed by the Westat IRB and received expedited initial approval on January 28, 2010 (see attachment 6).
Expected response rates are approximately 15 to 25%. The number of questions is kept to a minimum to limit the time needed to complete the survey and encourage response. In addition, responses are multiple choice using a Likert scale of 1 to 5 that also minimizes the time needed to complete the survey. Followup e-mails are sent to participants that have not responded, and a gift card giveaway is used to encourage participation. No follow up is completed for non-respondents.
Surveys are distributed to a small number of staff for review prior to distribution to the target population. No pre-test or sampling is completed for the surveys.
No consultations with persons outside of the CTEP contracting team or the contractor have been made for the purposes of the surveys. The survey team is lead by the CTSU Project Director, Steve Riordan, and the CTSU Quality Assurance Coordinator, Khalil Ayubi. The implementation team consists of the CTSU Customer Service Manager, the CTSU Promotion, Education and Training Manager, the CTSU web site lead, and other CTSU staff members as assigned.
Steve Riordan CTSU Project Director Westat Telephone: 301-517-4045 Email: [email protected] |
Khalil Ayubi CTSU Quality Assurance Coordinator Westat Telephone: 240-453-5610 Email: [email protected] |
Lucille Patrichuk CTSU Customer Service Manager Westat Telephone: 240-314-2572 Email: [email protected] |
Jenny Hopkins CTSU Promotion, Education and Training Manager Westat Telephone: 301-738-3556 Email: [email protected] |
Jayan Nair CTSU Web Site Lead Westat Telephone: 301-212-2126 Email: [email protected] |
|
| File Type | application/msword |
| File Title | TABLE OF CONTENTS |
| Author | Vivian Horovitch-Kelley |
| Last Modified By | Vivian Horovitch-Kelley |
| File Modified | 2010-08-19 |
| File Created | 2010-04-12 |