SUPPORTING STATEMENT A For:
A Generic Submission for Formative Research, Pretesting, and Customer Satisfaction of NCI’s Communication and Education Resources (NCI)
This submission is a combination of a currently approved generic submission:
“Pretesting of NCI’s Office of Communication Messages”
(OMB No. 0925-0046, Expiration Date 1/31/2010)
and
a formerly approved generic submission,
“Customer Satisfaction with Educational Programs and
Products of the National Cancer Institute”
(OMB No. 0925-0526, Expired 2/28/2007)
Yellow text identifies revisions from the #0925-0046 2008 Submission
December 30, 2009
Nina Goodman, MHS
Senior Advisor
Office of Market Research and Evaluation
Office of Communications and Education (OCE)
National Cancer Institute/NIH
6116 Executive Blvd., Suite 400
Rockville, MD 20892-8351
Phone: 301-435-7789
Fax: 301-480-3441
Email: [email protected]
Table of Contents
A. Justification 1
A.1 Circumstances Making the Collection of Information Necessary 1
A.2. Purpose and Use of the Information 4
A.3 Use of Improved Information Technology and Burden Reduction 6
A.4 Efforts to Identify Duplication and Use of Similar Information 10
A.5 Impact on Small Businesses or Other Small Entities 11
A.6 Consequences of Collecting the Information Less Frequently 11
A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 11
A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency 12
A.9 Explanation of Any Payment or Gift to Respondents 14
A.10 Assurance of Confidentiality Provided to Respondents 15
A.11 Justification for Sensitive Questions 17
A.12 Estimates of Hour Burden Including Annualized Hourly Costs 18
A.13 Estimate of Other Total Annual Cost Burden to Respondents or
Record Keepers 21
A.14 Annualized Cost to the Federal Government 21
A.15 Explanation for Program Changes or Adjustments 22
A.16 Plans for Tabulation and Publication and Project Time Schedule 23
A.17 Reason(s) Display of OMB Expiration Date is Inappropriate 25
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 25
Attachments
ATTACHMENT 1: Explanation of Types of Research
Attachment 2: Data Collection Methods Used in Previous Sub-studies
ATTACHMENT 3: Generic Sub-study Descriptions
OMB No. 0925-0046 conducted between 2007-2010; and
OMB No. 0925-0526, conducted between 2004-2007
ATTACHMENT 4: Memo from NIH Privacy Act Officer
ATTACHMENT 5: Memo from NIH Office of Human Subjects Research (OHSR)
JUSTIFICATION
A.1. Circumstances Making the Collection of Information Necessary
The National Cancer Institute’s (NCI) Office of Communications and Education’s (OCE) Office of Market Research and Evaluation OCE's Office of Market Research and Evaluation (OMRE) conducts various types of research to identify and learn about target audiences.
Specifically, OMRE conducts market and user-centered research and evaluation in order to:
Identify perceived cancer-related needs of diverse audiences
Inform the design and development of NCI resources and ensure that they are appropriate and effective, and reach the intended audiences
Monitor audience trends
Build on and advance the Institute's knowledge base and positioning in the field of user-centered informatics research
Assess the impact of resources and activities
The research conducted in OMRE helps ensure that NCI communication and education resources are appropriate, useful, and effective. This NCI office is requesting OMB to review this package which combines and streamlines two approved generic submissions into a single request. The two generic submissions are: “Pretesting of NCI’s Communications Messages” (OMB #0925-0046; expiration: 1/31/2010) and “Customer Satisfaction with Educational Programs and Products of the National Cancer Institute (OMB# 0925-0526; expiration 2/28/07).
For context and historical purposes, NCI would like OMB to know that since the approval of these OMB packages, NCI has experienced reorganization of their communications offices. The “Formative Research and Pretesting” information collection originally resided within the former Office of Communications (OC) within NCI’s OD, and the “Customer Satisfaction” information collection originally resided within the “Office of Education and Special Initiatives” (OESI) within NCI’s OD. Those two offices were combined in 2007 to form the current NCI Office of Communications and Education (OCE). As the two functions that previously oversaw these packages have been combined into one office, this current proposal represents the combination of the two previous packages.
The National Cancer Institute (NCI), established under the National Cancer Act of 1937, is the Federal Government's principal agency for research on cancer cause, prevention, detection, diagnosis, treatment, and rehabilitation, and for the dissemination of information for the control of cancer. Current authorization for NCI's education and information dissemination activities is contained in Section 410 of the Public Health Service Act (42 USC § 285a-2).
NCI’s OCE is the main office within the NCI that provides communications expertise within the Institute and between NCI and a variety of organizations and audiences, including Congress, other executive agencies, state and local governments, scientific and medical communities and institutions, voluntary groups, the press, the general public, and cancer patients. The OCE, which supports NCI’s priorities through activities that span NCI programs, specializes in the design, implementation, and evaluation of education programs over the entire cancer continuum, including prevention, screening, diagnosis, treatment, survivorship, and palliative care. The office also manages NCI initiatives that address specific challenges in cancer research and treatment. Health care providers, professional societies, patient groups, federal agencies, and the public are audiences for OCE’s educational programs and products/materials.
Within OCE, the Office of Market Research and Evaluation (OMRE) ensures that the National Cancer Institute’s (NCI) communication and education resources are appropriate, useful, and effective. OMRE uses scientific methods to: identify the cancer-related needs of diverse audiences; inform the design and development of NCI resources; monitor audience trends; and, assess the impact of resources and activities.
Information programs and campaigns within NCI create and use a variety of media including print (e.g., brochures, posters, fact sheets, information kits), broadcast (e.g., public service announcements, video news releases), and electronic formats (e.g., Internet, listservs), as well as direct response (Cancer Information Service) to inform and educate the public and health professionals about cancer. Production of these materials is the major way that the Institute relays messages to the audiences it is mandated to reach.
It is OCE’s responsibility to ensure that the materials produced by NCI can be understood and are well-received by intended audiences (Formative Research/Pretesting), and meet the satisfaction of NCI’s audiences (Customer Satisfaction). Formative research/pretesting helps ensure that messages have the potential to be received, understood, and accepted by those for whom they are intended; while the customer satisfaction research helps NCI ensure the relevance, utility, and appropriateness of the many educational programs and products that the Office produces. See Attachment 1 (Explanation of Types of Research) for additional details on the purpose of and distinctions between the two types of research. Also, Attachment 2 (Data Collection Methods Used in Previous Studies) contains a listing of studies approved and implemented under OMB 0925-0046 and under OMB# 0925-0526, along with each study’s methodology.
Through this submission, OCE is requesting generic clearance to: 1) continue conducting formative research and pretesting activities; and 2) resume collecting information to assess customer satisfaction with those products, with a renewal generic clearance package for the next three years.
A.2. Purpose and Use of the Information
Formative research, pretesting, and the assessment of customer satisfaction are integral parts of NCI’s overall plan for its communications strategies. Pretesting of print, broadcast, and web products and information services assesses the potential effectiveness of these products while they are still in the developmental stage. Given the large number and wide spectrum of cancer education programs and products OCE and NCI develop and disseminate, NCI requires accurate, timely, and useful information about the relevance, usefulness, and appropriateness of these products to its customers.
The formative research process is used to determine whether or not a draft message or message concept is effective in reaching and communicating with its audience. Pretesting involves presentation of draft messages designed to convey specific information to a sample of the audience for whom the materials are intended. These respondents are asked to give their reaction to the messages through either individual or group interviews. Information collected to determine the level of customer satisfaction with products helps NCI identify strategies for improving the accessibility of materials/programs, their user-friendliness, and their relevance to the needs of cancer patients and their families, health educators and interventionists, cancer advocates, cancer information specialists, and health care professionals. Research explores various domains shown in the table below.
Factors assessed |
Research questions |
Attention |
Do the messages attract and/or hold the audience's attention? |
Comprehension |
Are the messages and language clearly understood, and does the main theme of the message resonate with the audience? |
Personal relevance and Self-efficacy |
Do members of the target audience perceive the message as personally relevant, and do they see themselves as capable of acting on the message provided? |
Believability |
Is the message and/or its source perceived as credible? |
Acceptability |
Is there anything in the message that may be offensive or unacceptable to the target audience, and is the piece culturally sensitive? |
Accessibility |
Will members of the target audience be able to find the information or message with relative ease, considering factors such as the availability of products in multiple formats (e.g., print materials, videos, online documents, CDs, audio tapes) and in a variety of settings (e.g., clinics, hospitals, doctors’ offices, community organizations, libraries)? |
Usability |
How likely is the respondent to use the information in the format provided, and is the content provided in a logical, organized and user-friendly way? And, do the topics addressed by NCI and OCE products meet the needs, concerns and interests of its customers? |
Behavioral intent |
Do respondents think they will take action as a result of seeing/hearing the message? |
Readability and clarity |
Can the reader understand publications and educational materials, both in terms of the needs of low-literacy audiences and with respect to plain language principles and design? |
Availability |
Are NCI and OCE products present for low-English fluency audiences? |
Cultural appropriateness |
Are foreign-language translations or adaptations of products accurate and available for customers? |
Respondents' input and reactions to each of these areas provide insight into how the audiences for these messages may react, how the messages should be formulated or revised to communicate most effectively, and the usefulness, relevance and appropriateness of these products to its customers.
Other information that may be gathered on respondents regarding gender, age, socioeconomic level, race/ethnicity, and family medical history provides a basis for evaluating whether the messages may be perceived differently by different segments of the audience. For example, selected age groups may find a particular brochure or message on cancer prevention more relevant than other age groups.
Systematic formative research and pretesting has been widely adopted by health education program planners as an integral step in the development and targeted dissemination of messages and materials. Through this research, NCI is able to:
Understand characteristics of the target audience—its attitudes, beliefs, and behaviors—and use these in the development of effective communications tools;
Design messages and select formats that have the greatest potential to influence the target audience’s attitudes and behavior in a favorable way;
Determine the best promotion and distribution channels to reach the target audience with appropriate messages;
Expend limited program resource dollars wisely and effectively; and
Results of past pretesting and customer satisfaction efforts have been instrumental in helping NCI carry out its legislative mandate. A few examples of formative/pretesting efforts which have resulted in the development and refinement of NCI messages, materials, strategies, and formats (under OMB No. 0925-0046, expiring 1/31/2010), as well as customer satisfaction efforts which have resulted in NCI determining users’ level of satisfaction with products and identifying strategies for improvement (OMB No. 0925-0526, expired 2/28/07) are described in Attachment 3 (Previous Sub-study Descriptions).
A.3. Use of Information Technology and Burden Reduction
Information collection may be conducted using a variety of methodologies and technologies, such as one-on-one interviews, group interviews, or self administered questionnaires, depending upon the target audience being questioned and the subject matter being addressed.
As computer technology has continued to improve and become more widespread, opportunities to pretest messages on the Internet using either Web site questionnaires or on-line focus groups with Internet users have increased. Improved technology in the collection and processing of data has the potential to reduce the time burden for respondents and data collectors. For example, respondents can access and respond to data collection requests at a time and place that is convenient to them, eliminating the need to travel for in-person or group interviews. Wherever possible, NCI will make use of Web- or computer-based data collection methods. Transmission of data collection instruments and responses by electronic mail or facsimile will be utilized as appropriate (for example, with intermediary audiences such as NCI’s Cancer Information Service or Public Affairs Network). NCI anticipates that of the majority of data will be collected electronically.
Possible information technologies to assess customer satisfaction include
computer-assisted telephone interviewing (CATI), computer-assisted personal interviewing (CAPI), audio and computer-assisted self-interviewing (ACASI), on-line surveys, and focus groups. Each technology/method is briefly described below:
CATI’s technological capabilities include automated dialing, scheduling unanswered calls or interrupted interviews for efficient callbacks, random selection of respondents, automated skip patterns, instantaneous out-of-range checks, insertion of information from one question to guide a subsequent question, and the automated generation of databases for subsequent analysis. CATI will be utilized when geographic diversity is important and participants come from hard-to-recruit populations, such as physicians or Native Americans. CAPI technology allows interviewers to ask questions of a respondent using a computer to enter data. ACASI software technology offers many advantages of CAPI technology, but removes the need to have a person administer an interview. Instead, survey questions are pre-recorded and played back through the sound system of a computer, which the respondent can listen to privately by using headphones. Respondents select an answer by pressing a key that corresponds to one choice shown on the screen, after which answers are fed directly into a computer database.
ACASI surveys can also be administered over a telephone by entering the response on the telephone keypad. ACASI technology is particularly useful in administering surveys to low-literacy populations or when addressing sensitive topics that respondents may not feel comfortable discussing in the presence of someone else.
On-line surveys represent an especially convenient option for eliciting feedback from consumers of Web-based products. Respondents complete online surveys regarding a product and then submit the data electronically over the Internet. With online surveys, respondents can easily submit feedback during or immediately after using a Web-based product. They also allow participation from international audiences with virtually no additional costs.
Technology now enables the conduct of focus groups, which traditionally take place in-person with a moderator facilitating a discussion regarding a product, issue, or program with a small group of individuals in a designated physical location. Depending on factors such as geographic distribution and schedules of NCI customers and on the nature of the products and services under investigation, focus groups, when appropriate, may be implemented using a variety of technology-based formats:
Videoconferencing uses video and satellite technology to allow a group of participants located in multiple geographic locations to interact with one another both visually and aurally. A facilitator and a technical team located in a hub site maintain the video and audio connections among participating sites.
Internet or online conferencing is especially useful for discussions with specific individuals or international participants. This format functions as a sort of “chat room” in which a moderator intercepts and distributes e-mail transmissions from participants who have logged onto a specially designated Web site.
Teleconferencing uses telephone technology to facilitate an exchange among participants located in multiple geographic locations. Participants dial into a specially designated phone number or “bridge line” that is moderated by a focus group facilitator.
Advanced Strategies Lab (ASL) is a qualitative research process that uses an online interactive discussion. In the 90-minute, online ASL sessions, participants receive instructions and are asked questions from the moderator over the phone on a conference call. Participants’ answers are typed into their designated dialogue box (which hides their identity from the other participants to ensure confidentiality). The facilitator then guides the group through online brainstorming, discussion, and assessment exercises. The ASL’s software automatically formats and tabulates data as the session proceeds. Summaries of survey results are available seconds after respondents complete a question, and full verbatim reports are available within hours after a session.
A Privacy Impact Assessment (PIA) is being undertaken with the NCI Privacy Act Coordinator to assess the security of the IT systems and mitigate any risks.
A.4. Efforts to Identify Duplication and Use of Similar Information
The areas in which information needs to be gathered (as described in A.2. above - attention, comprehension, etc.) to pretest effective cancer messages (brochures, PSAs, media campaigns, website, etc.) are generally similar from pretest to pretest. However, since the cancer field is so diverse and complex, and each message is essentially different, new data collection instruments generally must be prepared for each pretest. As each new message, strategy, or product is developed, NCI reviews existing literature and data bases, including pretesting reports on existing messages and materials, and consults with outside experts to evaluate available information on similar messages with comparable audiences.
NCI will continue to assess other NIH active generic ICs upon submission of applications to conduct each sub-study. Currently, the existing NIH active generic ICs are very site- or method-specific. Site-specific generic customer satisfaction ICs at NIH are located at the Clinical Center (OMB No. 0925-0458), Center for Scientific Review (0925-0474), and National Library of Medicine (OMB No. 0925-0476). Method- or content-specific generic ICs at NIH include customer satisfaction surveys regarding the procedures and application and awards of grants (OMB No. 0925-0534) or the customer satisfaction of NIH internet sites (OMB No. 0925-0486). At this time, the above-listed generic ICs listed would not be able to accommodate the population and content that the National Cancer Institute’s generic IC anticipate needing in the areas of customer satisfaction and pretesting NCI communication messages.
Additionally, NCI has an internal review process for surveys that will be used by this generic to assess the quality of each survey prior to its use. The NCI will provide direct oversight for any and all surveys conducted under this generic clearance to avoid duplication of effort and information collected.
A.5. Impact on Small Businesses or Other Small Entities
Small businesses that are non-profits and independently-owned may be participants in this generic submission. The small businesses we may include are physicians, health care providers, and highly specialized individuals may sometimes be the target audience for NCI’s communication information and customer satisfaction materials. When small businesses are asked to complete an information collection, all efforts will be made to reduce their burden by using a short questionnaire/survey and interviewing fewer small businesses than larger ones.
A.6. Consequence of Collecting the Information Less Frequently
For the most part, formative research, pretesting, and customer satisfaction information will be collected only one time for each print, broadcast, or electronic message, product/material, or strategy tested. However, there may be occasion where a pre- and post-test to assess differences in communication and/or satisfaction messages may be useful for a particular sub-study. Additionally, previous respondents may be contacted to participate in follow-up studies if they have originally granted consent for such and if the subsequent study uses that population.
A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
NCI recognizes the need to collect information in a manner that places minimal burden on each respondent. Therefore, NCI anticipates that information collections under this clearance package will comply with 5 CFR 1320.5(d)(2) requirements with only two anticipated exceptions.
When NCI requires responses to a self-administered written questionnaire in less than 30 days, receipt of the questionnaire is generally preceded by advance notification to respondents explaining the purpose of the questionnaire, the approximate length of time that the questionnaire will take, and the voluntary nature of participation. All efforts are made to keep such questionnaires short and focused.
Because NCI’s pretesting activities are often qualitative in nature, the results are not generalizable to the population at large or to the particular target audience under study. However, the nature of pretesting is such that generalizability is not a critical feature; the emphasis is on obtaining timely, useful information that can be fed back into the development of new messages or materials or the revision of existing ones.
There are no other special circumstances.
A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside
Agency
The 60-Day Federal Register notice soliciting comments on this study prior to initial submission to OMB was published on October 28, 2009, Vol. 74, p. 55558-55559. One public comment was received on October 28, 2009, in regards to NCI communication. An email response was sent on October 28, 2009, stating, “We received your comment. We will take your comments into consideration”.
NCI, along with other Public Health Service agencies, has been a leader in the development of methods for developing, testing, and disseminating health information. The work of many advisers over many years has brought us to where we are today. A number of outside health communications experts were consulted to review the plans contained herein for program development research and evaluation of NCI communications programs and their comments and suggestions have been incorporated into these data collection plans.
Although all consumer materials, as well as materials produced for the media, must go through DHHS approval procedures, NCI is not required to coordinate with other Federal agencies in its education, information dissemination, and evaluation activities. Nevertheless, NCI has consulted with other agencies in the past and will continue to do so in the future, as appropriate, to help ensure accuracy and consistency, and to avoid duplication of effort. It should also be noted that many of the pretesting and formative research efforts conducted by NCI have been requested and used by outside agencies to inform their own communications activities. NCI has worked with other agencies in the past (as evidenced by the submission previous to this one), although this did not occur for any submissions during the last three years. OCE also consults with several internal experts across NCI, most frequently in NCI’s Division of Cancer Control and Population Sciences (DCCPS) (e.g., Rick Moser, PhD and Gordon Willis, PhD) on the conduct of study design, methods, and data analysis and reporting.
Richard P. Moser, Ph.D.
Research Psychologist
Office of the Associate Director
Behavioral Research Program
Division of Cancer Control and Population Sciences
National Cancer Institute
6130 Executive Blvd., MSC 7326
Executive Plaza North, Room 4052
Rockville, MD 20852
(301) 496-0273 (voice)
Gordon Willis
Cognitive Psychologist
Applied Research Program
Division of Cancer Control and Population Sciences
National Cancer Institute
6130 Executive Blvd, MSC 7344
EPN 4005
Bethesda, MD 20892-7344
A.9. Explanation of Any Payment or Gift to Respondents
It is possible that some information collection activities will entail small payments or gifts to respondents. Small amounts of money, a free meal or snack scheduled around the time of the pretest, and/or renumeration for parking and/or transportation are most often used, particularly when recruiting hard-to-reach and minority respondents. Renumeration may also be necessary in some sub-studies where highly specialized individuals are invited to participate in a survey, and will cover the cost of transportation and other types of local expenses.
Research has shown the advantages of providing a small incentive for improving response rates and decreasing item nonresponse, especially in mail and telephone surveys.1 Studies of participants in the original National Health and Nutrition Examination Surveys (NHANES) found that response rates for those told they would receive remuneration versus not were 82% and 70%, respectively.2 The National Survey of Family Growth conducted an experiment with remuneration of respondents and found that incentives increased response rates, reduced interviewer labor (broken appointments and callbacks), and improved data quality.3 In the 1999 Observing Protein and Energy Nutrition study, remuneration was credited as contributing to high response and retention rates.4
Instances for offering a small incentive will be determined on a case-by-case basis (depending on the particular information collection design). The following are the kinds of situations for which respondents may be paid or given a gift:
Physicians who participate in a survey on their opinions about the relevance of materials for their patients may receive a small payment (typically $25-50, but up to $80, depending, on market) or donation to a charitable organization in their honor.
Individuals who participate in in-person focus groups may receive an honorarium (perhaps $25-50*) to cover their time, transportation costs, and childcare expenses.
Health educators who support the acquisition of data related to customer satisfaction with specific educational programs and products may be able to request and receive certain quantities of materials that exceed the limits usually established for those materials.
Circumstances, however, do not always require that renumeration be given; many audiences including the public, patients, survivors, and other health professionals often participate gratis because of their interest or involvement in the topic, or as a professional courtesy. For example, in situations when the general public is completing an online survey, no remuneration will be involved unless influenced by other factors.
*Qualitative research facilities in several major markets have ensured NCI that they will not be able to recruit focus group participants, particularly health care providers, if the remuneration is less than $80.
A.10. Assurance of Confidentiality Provided to Respondents
Information provided by respondents will be kept confidential and private, except as otherwise required by law. This will be communicated to respondents by means of introductory letters, explanatory texts on the cover pages of questionnaires, scripts read prior to focus groups or telephone interviews, and consent forms. Respondents will also be advised of the following: the nature of the activity; the purpose and use of the data collected; NCI sponsorship; and the fact that participation is voluntary at all times. Because responses are voluntary, respondents will be assured that there will be no penalties if they decide not to respond, either to the information collection as a whole or to any particular questions.
As a further guarantee of confidentiality, all presentation of data in reports will be in aggregate form, with no links to individuals preserved. Reports will be used only for research purposes and for the development of communication messages and educational materials. Only NCI staff and contractor personnel conducting the information collection will have access to individual-level survey, interview, or focus group data. All project/contractor staff conducting the information collection will sign a confidentiality agreement, and all electronic and hard-copy data will be maintained securely throughout the information collection and data processing phases. While under review, electronic data will be stored in locked files on secured computers; hard-copy data will be maintained in secure building facilities in locked filing cabinets. Before any data are released for public use data sets, any identifying information will be stripped from each respondent’s record and the identifying information will be destroyed.
The NIH Privacy Act Officer has reviewed the work scope of this proposal to determine whether the Privacy Act is applicable to this data collection and will be asked to review the protocol of each collection under this blanket clearance to ensure that NCI adheres to privacy requirements (see Attachment 4, Memo from NIH Privacy Act Officer).
Although some personally identifiable information will be collected, data will not be retrieved by personal identifiers unless the respondent voluntarily agrees to provide the information, so he/she can be contacted for follow-up (in rare situations). Instances could arise for activities that, for example, gather and retain respondent names and contact information (such as addresses, telephone numbers, or email addresses). This information would be used to measure customer satisfaction information regarding an NCI/OCE product at one time and then at some point later (for example, to learn whether customers have gone back to the product for additional review or information.)
Before submitting each sub-study for OMB review, NCI always submits related IRB paperwork to the Office of Human Subjects Research (OHSR). However, pretesting efforts described in this proposal are typically considered exempt from the “Regulations for the Protection of Human Subjects,” in accordance with paragraph (b)(3) of 45 CFR Sec. 46.101 (http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.htm), and as deemed by the OHSR (see Attachment 5, Memo from NIH OHSR). OCE understands and has experience with the process and plans to continue submitting for review/exemption, using Exemption Request forms found here: http://ohsr.od.nih.gov/info/info.html; http://ohsr.od.nih.gov/info/pdf/requestforReview.pdf. (Note: Please contact Nina Goodman with questions).
A.11. Justification for Sensitive Questions
As mentioned in sections A.2. and A.10. above, some studies require the inclusion of people who match selected characteristics of the target audience that NCI is trying to reach. Therefore, personally identifiable information (PII) such as race/ethnicity, income, education and/or medical/health status, are required to be asked on the initial screening questionnaire used for recruiting. Potential participants are informed that this is being done to make sure that NCI speaks with the kinds of people for whom its messages are intended. Again, respondents are assured that the information is voluntary and will be treated as confidential. All information on race/ethnicity will comply fully with the standards of OMB Statistical Policy Directive No. 15, October 1997. (http://www.whitehouse.gov/omb/fedreg/1997standards.html).
Since NCI communications are concerned with the detection, diagnosis, treatment, and prevention of cancer, some projects may involve asking questions about (or discussing) how one perceives his/her own personal risk for serious illness. Fears of cancer and experiences with cancer may also be covered. This information is needed to gain a better understanding of the target audience so that the messages, strategies, and materials designed will be appropriate and sensitive. Questions of this nature, while not as personal as those on sexual behavior or religious beliefs, still require some sensitivity in how they are worded and approached. In face-to-face data collections, questions of this kind are generally asked later in the interview or group discussion when respondents are more comfortable with the interview situation and are more at ease with the interviewer/moderator. As noted in section A.10., participants are informed in advance about the nature of the activity and the voluntary nature of their participation. The interviewer/moderator makes it clear that they do not have to respond to any question that makes them uncomfortable, and that no questions of a sensitive nature will be asked in the course of collecting information from respondents.
Raw data from data collections that include sensitive information (for example, screening questionnaires and audio tapes) are not retained once the data have been extracted and aggregated; nor does the information become part of a system of record containing permanent personally identifiable information that can be used for retrieval.
A.12. Estimates of Hour Burden Including Annualized Hourly Costs
The number of respondents to be included in each communications research and customer satisfaction sub-study will vary, depending on the target audience and the nature of the message/materials/program being assessed. Samples could be small or large, and burden per respondent can range from a few minutes to 90 minutes. It is also difficult to pinpoint the actual number of studies required during the next three years; we can only anticipate need based on past studies. For illustrative purposes, Table A.12-1 below provides estimations of a distribution of respondents and hours over three years by type of data collection method. Even if the actual distribution of interviews, surveys, and focus groups varies from the estimate below, the total burden will not be exceeded without additional OMB approval. The proposed data collection methodologies are described in more detail in Attachment 1 and Supporting Statement B. Annual burden, therefore, is one-third of the total figures presented here. It is estimated that on an annual basis, the total number of respondents is 11,000 and the annual estimated burden is 2,350 hours.
Table A.12-1 Estimates for Burden Hours For Three Years (Generic Study)
Survey Method |
Total Number of Respondents |
Frequency of Response |
Minutes/Hour Per Response |
Total Burden Hours |
Focus Groups |
900 |
1 |
90/60 (1.5) |
1,350.0 |
Individual In-Depth Interviews (Typically longer than 15 minutes, includes website usability testing) |
600 |
1 |
45/60 (.75) |
450.0 |
Brief Interviews (Typically less than 5 minutes) |
19,000 |
1 |
10/60 (.166667) |
3,166.7 |
Surveys (Web, phone, in-person, paper-and-pencil) |
12,500 |
1 |
10/60 (.166667) |
2,083.3 |
Totals |
33,000 |
|
|
7,050.0 |
*On occasion, respondents may give permission for a follow-up survey or interview. As this happens fairly infrequently, we are not adding line for interview or survey respondents who give two responses. Sub-study submissions will indicate when NCI anticipates follow-up surveys/interviews with a given number of respondents.
Table A.12-2 presents the approximate cost to respondents over the 3-year life of the project. Annual cost, therefore, is one-third of the total figures shown. The cost to individual respondents who are members of the general public is approximately $3.40/response based on the estimate of $20.00/hour and an average respondent burden of .17 hours per respondent. This rate of reimbursement is a generally accepted one in the market research industry. While physicians (general practitioners) sometimes participate gratis in telephone or self-administered surveys (time permitting), it is customary to reimburse them at the average rate of $80.00 per hour for taking part in focus groups (http://www.bls.gov/oes/current/oes291062.htm). Assuming one focus group study with physicians per year (6 groups of 8 physicians each for 1 ½ hours), the annual cost would be almost $6,000.00. Non-physician health professionals and researchers will be reimbursed at the rate of $35/hour and $31/hour, respectively. (These rates were determined using May 2008 data from the website of the Bureau of Labor and Statistics; http://www.bls.gov/oes/current/oes_nat.htm).
Additionally, the costs assume that the general public will only be completing focus groups, and the physicians completing individual in-depth interviews, etc. however, this very well may not be the case. It is difficult to estimate the number of each type of respondents that will complete different survey methods and for the ease of calculation, it was calculated as such. Therefore, the costs may increase or decrease, based on this assumption.
A.12-2 Cost to Respondents For Three Years (Generic Study)
Type of Respondents |
Total Burden Hours |
Hourly Respondent Wage Rate |
Cost |
General Public (patients, survivors, family members) |
1,350.00 |
$20.00* |
$27,000.00 |
Physicians (GPs, oncologists, radiologists) |
450.00 |
$80.00 |
$36,000.00 |
Health Professionals (non-physicians: NPs, Pas, Nurses) |
3,166.67 |
$35.00 |
$110,833.45 |
Researchers |
2,083.33 |
$31.00 |
$64,583.23 |
TOTAL |
7,050.00 |
|
$238,416.57 |
*Hourly wage of $20.00 per hour (rounded) is based on the mean U.S. hourly wage, May 2008, Bureau of Labor Statistics (BLS). Hourly wages for researchers and health professional categories are also averages based on data from the BLS.
A.13. Estimate of Other Total Annual Cost Burden to Respondents or Record Keepers
There are no capital or start-up costs to the data collection efforts requested; nor are there any costs associated with operation, maintenance or purchase of services.
A.14. Annualized Cost to the Federal Government
The estimated annualized cost to the federal government is approximately $361,380, which amounts to a total estimated cost of $1,084,140 for a period of three years. Table A.14-1 shows the breakdown of estimated costs, with figures that include the costs of study design, facility rental (e.g., for focus groups), data collection, analysis, and report/publication writing. This estimate also includes monitoring by the Government Project Officer and involvement by NCI's Senior Analyst and supporting staff, projected to be about 500 hours of effort a year. Given a total NCI personnel average cost of $60.00 per hour, $30,000 would be spent annually on Government staff salaries (or $90,000 over the 3-year period, which is reflected in the $361,380 annual total.
Table A.14-1 Annualized Costs to the Federal Government
|
|
|
Estimated costs* |
Annual costs for NCI staff to plan, conduct, and analyze the outcomes of information collection activities:
|
Managerial |
FTE |
$3,000 |
Professional |
FTE |
$25,000 |
|
Support |
FTE |
$2,000 |
|
IN-DEPTH INTERVIEWS
|
$55,440 |
||
FOCUS GROUPS
|
$105,840 |
||
INTERVIEWS - approximately 3 central location interview studies at $11,340 each - approximately 10 telephone interview studies at $3,780 each - approximately 4 gatekeeper reviews (interviews) at $11,340 each
|
$34,020 $37,800 $45,360 |
||
SURVEYS - approximately 4 self-administered questionnaire studies at $5,670 each - approximately 2 telephone omnibus surveys at $15,120 each
|
$22,680 $30,240 |
||
TOTAL |
$361,380 |
||
*All costs are estimates based on costs for past research conducted under the previous package.
A.15. Explanation for Program Changes or Adjustments
This information collection request is considered a program change. This revision combines and streamlines two approved generic submissions into a single request. The currently approved generic submission is: “Pretesting of NCI’s Communications Messages” (OMB #0925-0046; expiration: 1/31/2010) and the formerly approved submission, “Customer Satisfaction with Educational Programs and Products of the National Cancer Institute (OMB# 0925-0526; expiration 2/28/07). These collections were previously conducted by two offices that have since merged to become one, the Office of Communications and Education (OCE). Now that they have merged and these collections are so intertwined, it is logical that both types of collections be submitted under the same package.
When combined, the total number of respondents has decreased from the prior packages. This is because the brief interviews with CIS callers in the previous submission are no longer needed. Additionally, the total burden hours has increased by about 15%, probably as a result of the increased respondents estimated to do focus groups, which amounts for the highest hours per response, as well as the increase in ease of responding to web-based surveys. There is also an increase in costs due to inflation as well as an increase in labor costs associated with carrying out data collections of this kind.
A.16. Plans for Tabulation and Publication and Project Time Schedule
Evaluation/research staff will review the material to be tested; discuss the objectives with the individuals responsible for development; determine the analytic questions to be addressed; and then prepare the research procedures, instruments, and data analysis plan. The analysis conducted for each study will be determined by the objectives of the research, the messages being tested, and the audience for the messages. Specifics of each analysis cannot be determined until the messages to be tested are prepared. Research techniques may include qualitative analysis, customer satisfaction analysis, descriptive statistics, statistical analysis and parametric statistical tests. Analysis may be conducted using cross-tabulation procedures, with categorical variables; or between-group procedures, with continuous variables.
While the primary purpose of all studies is to provide information to the developers of the messages/materials/services for the purpose of improving them, NCI shares information internally and also makes results available to a variety of health program planners at Government agencies, voluntary organizations, health professional organizations, and medical institutions. Information collected will be compiled and presented in reports and briefings for staff from OCE and other divisions within NCI and NIH. Reports will include information regarding respondent demographics, basic descriptive data with OCE products and services, comparisons across demographic and customer subgroups, recommendations for improving programs and products, and analyses of longitudinal changes. In addition, NCI may also publish results in journals and present the findings of its research at meetings of professional associations, for example, the American Public Health Association and the Society for Public Health Education. Formative research conducted by OCE is also sometimes summarized in news-related publications such as the NIH Record.
The specific messages and materials/service that will be evaluated and the timing of these studies are not known at this time. While the research period varies somewhat depending on the complexity of the testing and number of respondents required, the typical study will require approximately 12 weeks from initial design to preparation of the report of pretest findings. A schedule for a typical pretest is shown below:
A.16-1 Project and Publication Timeline
Activity Time Schedule
Initial review of research questions 1-2 weeks after OMB approval
Write data collection instrument 3-4 weeks after OMB approval
Preparation of design and plan 4-5 weeks after OMB approval
Review of design 6-7 weeks after OMB approval
Collection of data 7-8 weeks after OMB approval
Analysis of data 9-10 weeks after OMB approval
Write report of findings 11-12 weeks after OMB approval
Develop manuscript (when seeking publication) 4-6 months after OMB approval
Submit for publication 7-8 months after OMB approval
A.17. Reason(s) Display of OMB Expiration Date is Inappropriate
NCI will continue displaying the OMB control number and expiration date in the upper right-hand corner of all data collection instruments.
A.18. Exceptions to Certification for Paperwork Reduction Act Submissions
NCI is in full compliance with the provisions contained within the Certification for Paperwork Reduction Act.
1 E. Singer, J. Van Hoewyk, and M. P. Maher, “Experiments with Incentives in Telephone Surveys,” Public Opinion Quarterly, Vol. 64, No. 2, Summer 2000, pp. 171-188; A. H. Church, “Estimating the Effect of Incentives on Mail Survey Response Rates: A Meta-Analysis,” Public Opinion Quarterly, Vol. 57, No. 1, Spring 1993, pp. 62-79.
2 U.S. Department of Health, Education, and Welfare, “A Study of the Effect of Remuneration Upon Response in the Health and Nutrition Examination Survey, United States,” Vital and Health Statistics, Series 2-No.6, 1975.
3 W. D. Mosher, W. F. Pratt, and A. P. Duffer, “CAPI, Event Histories and Incentives in the NSFG Cycle 5 Pretest,” American Statistical Association, 1994 Proceedings of the Section on Survey Research Methods, Vol. 1, 1995, pp. 59-63.
4 A. F. Subar, V. Kipnis, R. P. Troiano, et al., “Using Intake Biomarkers to Evaluate the Extent of Dietary Misreporting in a Large Sample of Adults: the OPEN Study.” American Journal of Epidemiology, Vol. 158, 2003, pp. 1–13.
| File Type | application/msword |
| Author | McCarthyK |
| Last Modified By | NCI |
| File Modified | 2010-01-22 |
| File Created | 2010-01-22 |