SUPPORTING STATEMENT B For:
A Generic Submission for Formative Research, Pretesting, and Customer Satisfaction of NCI’s Communication and Education Resources (NCI)
This submission is a combination of a currently approved generic submission:
“Pretesting of NCI’s Office of Communication Messages”
(OMB No. 0925-0046, Expiry Date 1/31/2010)
and
a formerly approved generic submission,
“Customer Satisfaction with Educational Programs and
Products of the National Cancer Institute”
(OMB No. 0925-0526, Expired 2/28/2007)
Yellow text identifies revisions from the #0925-0046 2008 Submission
September 30, 2009
Nina Goodman, MHS
Senior Advisor
Office of Market Research and Evaluation
Office of Communications and Education (OCE)
National Cancer Institute/NIH
6116 Executive Blvd., Suite 400
Rockville, MD 20892-8351
Phone: 301-435-7789
Fax: 301-480-3441
Email: [email protected]
Table of Contents
B. statistical methods 1
B.1 Respondent Universe and Sampling Methods 1
B.2. Procedures for the Collection of Information 1
B.3 Methods to Maximize Response Rates and Deal with Nonresponse 4
B.4 Test of Procedures or Methods to be Undertaken 6
B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting
and/or Analyzing Data 7
Attachments
ATTACHMENT 1: Explanation of Types of Research
Attachment 2: Data Collection Methods Used in Previous Sub-studies (under OMB 0925-0046)
ATTACHMENT 3: Generic Sub-study Descriptions
OMB No. 0925-0046, conducted between 2007-2010; and
OMB No. 0925-0526, conducted between 2004-2007
ATTACHMENT 4: Memo from NIH Privacy Act Officer
ATTACHMENT 5: Memo from NIH Office of Human Subjects Research (OHSR)
B. statistical methods
B.1 Respondent Universe and Sampling Methods
The respondent universe consists of individuals who have used one or more of NCI’s materials, services, or programs. It is difficult to define and anticipate the types of potential respondents, but below are descriptions of the people who have and could represent NCI’s respondent universe in this generic submission:
People with cancer, their family members, and their caregivers
Physicians (general practitioners, oncologists, radiologists), nurse practitioners, physician assistants, nurses
Cancer researchers
As described in Supporting Statement A of this justification statement, the research that that will be conducted under this blanket clearance is not statistical research that calls for large numbers of survey respondents. NCI conducts these information collections for the purpose of program monitoring and improvement. As such, the sampling methods will not ensure a generalizability of results. The methods, however, will ensure that NCI collects enough data to inform decisions about messages, materials, programs, and services.
Office of Communications and Education (OCE) will ensure that each study population receives numerous requests to enable the highest response rate possible. Again, this will help guarantee that NCI is collecting enough data to make informed decisions about messages, materials, programs, and services.
B.2 Information Collection Procedures
Studies of formative research/pretesting and customer satisfaction with OCE and other NCI products may involve a wide range of information collection methods, including focus groups; as well as surveys administered using a variety of methods. Each information collection method is described below.
Focus Groups. Focus groups serve as a particularly useful medium to collect information from customers when rich, in-depth information regarding attitudes and reactions to products is desired. Focus groups traditionally take place in an in-person format, in which a moderator facilitates a discussion regarding a product, issue, or program. Depending on factors such as the geographic distribution and schedules of NCI customers and on the nature of the products and services under investigation, focus groups may use a variety of technology-based formats, including videoconferencing, Internet, and teleconferencing (see Supporting Statement, Section A.3. for a brief description of each).
Written surveys (mail surveys or questionnaires) are distributed to a sample of product customers through the mail. Respondents complete the questionnaires and return them for tabulation and analysis. Bounceback cards generally consist of short written questionnaires included with a product that the customer returns by mail after receiving and trying out the product, and serve as an effective means of eliciting quick-turnaround customer satisfaction feedback. Certain settings offer convenient venues for reaching a large audience who may be the intended recipients of NCI and OCE products. With intercept surveys, data collectors approach individuals in clinical or professional settings and ask them to complete written or oral questionnaires regarding a particular product.
Online surveys (see Supporting Statement A, Section A.3 for more information).
Telephone surveys are particularly useful for collecting information in relatively short time frames while maintaining high response rates, with resulting estimates generalizable to the universe of study. A sample of potential respondents is drawn and trained interviewers contact them to administer a questionnaire over the phone.
Computer-assisted telephone interviewing (CATI) surveys conducted by telephone are well-suited to this technology, which employs automated dialing; schedules unanswered calls or interrupted interviews for efficient callbacks; randomly selects respondents; uses automated skip patterns and instantaneous out-of-range checks; insertion of information from one question to guide a subsequent question; and the automated generation of databases for subsequent analysis.
In-person surveys, implemented in a central location or sometimes door-to-door by interviewers who canvass households or individuals living within a discrete geographic area to elicit information regarding certain topics or issues. Door-to-door surveys may be appropriate for the examination of customer satisfaction with OCE products and services that are targeted at persons living in specific neighborhoods or in smaller urban or rural communities, or persons who reside in geographic areas exposed to high health risks specifically addressed by certain OCE products. In-person data collection methods that incorporate CAPI and ACASI technology are described below:
Computer-assisted personal interviewing (CAPI) technology allows interviewers to ask questions of a respondent using a computer to enter data. Some primary advantages of CAPI include: the elimination of routing and looping problems within a paper-and-pencil questionnaire; respondents and interviewers cannot accidentally skip questions; interview questions are customized to account for personal information provided by the respondent (e.g., age, information from previous questions); software can automatically perform mathematical calculations and tabulations; software checks for inadmissible or inconsistent responses; allows interviewers to administer surveys to geographically isolated groups, respondents without telephones or Internet access, or other difficult-to-reach populations; CAPI eliminates errors that arise from separate data entry.
Audio and computer-assisted Self-Interviewing (ACASI) (see Supporting Statement A, Section A.3 for more information).
When necessary, we will ensure that industry standards are followed to ensure quality control. All data analysis will be conducted under the advice of an NCI statistician, as needed.
B.3 Methods to Maximize Response Rates and Deal with Nonresponse
In the case of data collection activities that involve interviews or telephone, mail, and in-person surveys, several procedures that have proven effective in previous studies will be used when possible to maximize response rates:
Potential respondents will be informed about the importance of these studies and encouraged to participate through a variety of methods, including newsletters from professional associations or community organizations, and letters of support from key individuals.
Experienced, highly-trained staff will moderate all focus groups and conduct all interviews and surveys.
Interviewers will participate in thorough training sessions. Training topics will include study objectives, question-by-question reviews of data collection instruments, strategies for engaging respondents, role playing, and techniques for fostering respondent cooperation and survey completion.
Well-defined conversion procedures will be established. If a respondent for a survey declines to be interviewed, a member of the contractor's conversion staff will contact the respondent to explain the importance of their participation. (Conversion staff is highly experienced telephone interviewers whose style and persuasive abilities have demonstrated success in eliciting cooperation. They receive a pay differential to acknowledge these skills, which also serves as an incentive to the interviewer pool, whose completion rates are carefully monitored to assess their qualifications to serve as conversion staff.)
For telephone interviews, outgoing calls that result in a disposition of no answer, a busy signal, or an answering machine will be automatically rescheduled for subsequent attempts. Up to 20 outgoing calls to a given number with dispositions of the sort listed will be made before declaring it a non-response.
Should a respondent interrupt an interview for any reason, such as needing to attend to a personal matter, the interviewer will reschedule or, in the case of telephone surveys, a predictive dialer will automatically reschedule the interview for completion at a later time.
Fielding for telephone and mail surveys will occur over a period of six or more weeks. Based on past experience, this time frame will allow the contractor to reach individuals who are on vacation, out of the home during irregular periods, have a temporarily disconnected telephone, or who are not answering the phone for some other reason.
Interview staff will be able to provide respondents with the name and telephone number of an official at NCI or OCE. This official will confirm with respondents the importance of their participation.
When appropriate, a dedicated toll-free number will be established at NCI or a contractor’s office to allow potential respondents to confirm a study’s legitimacy.
For mail surveys, a number of techniques will augment response rates:
A self-addressed, stamped return envelope will be enclosed with each survey.
Surveys will be mailed to respondents using stamps instead of metered postage labels.
Creative and attractive graphics will be used to attract the attention of respondents (e.g., different colored paper for successive survey iterations).
Hand-signed cover letters will be sent with each survey.
Follow-up mail or phone contacts will be made to encourage participation.
Respondents will be allowed the option of completing surveys online or faxing back completed surveys.
B.4 Tests of Procedures or Methods to be Undertaken
Before each information collection is implemented, contractors will pilot-test the instrument(s) and method of data collection. Lessons from the pilot test will be identified, and changes will be incorporated into the instrument and method, as necessary. All pretests will involve no more than nine individuals unless OMB clearance is sought for more than nine.
B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
The following individuals, among others, including contractors, who may be chosen to collect pre-testing and customer satisfaction information, will be responsible for the design of statistical and sampling procedures undertaken as part of these data collection activities:
Holly A. Massett, Ph.D.
Associate Director
Office of Market Research and Evaluation
Office of Communications and Education
National Cancer Institute
6116 Executive Blvd., Suite 400
Rockville, MD 20892-8351
National Cancer Institute
301-594-8193
Nina Goodman, MHS, Senior Advisor
Deputy to the Associate Director
Office of Market Research and Evaluation
Office of Communications and Education
National Cancer Institute
6116 Executive Blvd., Suite 400
Rockville, MD 20892-8351
301-435-7789
| File Type | application/msword |
| Author | McCarthyK |
| Last Modified By | Vivian Horovitch-Kelley |
| File Modified | 2009-10-19 |
| File Created | 2009-09-22 |