SUPPORTING STATEMENT A For:
A Generic Submission for Formative Research, Pretesting, and Customer Satisfaction of
NCI’s Communication and Education Resources
OMB No. 0925-0046, Expired 2/28/2013
This is a request for a reinstatement with change.
Yellow text identifies changes from the 2010 Submission.
April 8, 2013
Nina Goodman, MHS
Public Health Advisor
Office of Market Research and Evaluation
Office of Communications and Education (OCE)
National Cancer Institute/NIH
6116 Executive Blvd., Suite 400
Rockville, MD 20892-8351
Phone: 301-435-7789
Fax: 301-480-3441
Email: [email protected]
Table of Contents
A. Justification 1
A.1 Circumstances Making the Collection of Information Necessary 1
A.2. Purpose and Use of the Information Collection 4
A.3 Use of Improved Information Technology and Burden Reduction 6
A.4 Efforts to Identify Duplication and Use of Similar Information 9
A.5 Impact on Small Businesses or Other Small Entities 10
A.6 Consequences of Collecting the Information Less Frequently 10
A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 11
A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency 11
A.9 Explanation of Any Payment or Gift to Respondents 12
A.10 Assurance of Confidentiality Provided to Respondents 14
A.11 Justification for Sensitive Questions 16
A.12 Estimates of Annualized Burden Hours And Costs 17
A.13 Estimates of Other Total Annual Cost Burden to Respondents and
Record Keepers 19
A.14 Annualized Cost to the Federal Government 19
A.15 Explanation for Program Changes or Adjustments 20
A.16 Plans for Tabulation and Publication and Project Time Schedule 21
A.17 Reason(s) Display of OMB Expiration Date is Inappropriate 22
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 22
Attachments
ATTACHMENT 1: Explanation of Types of Research
Attachment 2: Generic Sub-studies and Information Collection Methods Used in 0925-0046 from 2010 to current
ATTACHMENT 3: Privacy Impact Assessment
ATTACHMENT 4: Memo from NIH Privacy Act Officer
ATTACHMENT 5: Form for NIH Office of Human Subjects Research Protection (OHSRP)
This is a request for OMB to approve this reinstatement with change request for this generic submission. As part of NCI’s mandate from Congress to disseminate information on cancer research, detection, prevention, and treatment, the Institute develops a wide variety of messages and materials. Testing these messages and materials assesses their potential effectiveness in reaching and communicating with their intended audience while they are still in the developmental stage and can be revised. The formative research and pretesting process thus contributes to maximizing NCI’s limited dollar resources for information dissemination and education. NCI also must ensure the relevance, utility, and appropriateness of the many educational programs and products that the Institute produces. Customer satisfaction studies help NCI identify modifications necessary to meet the needs of NCI’s various target audiences. Since the previous submission, there have been 22 approved sub-studies with an approved request of just under 3,000 burden hours over 2.5 years. Approval is requested for the conduct of multiple studies annually using such methods as interviews, focus groups, and various types of surveys. The content, timing, and number of respondents to be included in each sub-study will vary, depending on the nature of the message/material/program being assessed, the methodology selected, and the target audiences.
JUSTIFICATION
A.1. Circumstances Making the Collection of Information Necessary
The National Cancer Institute’s (NCI) Office of Communications and Education’s (OCE) Office of Market Research and Evaluation OCE's Office of Market Research and Evaluation (OMRE) conducts various types of research to identify and learn about target audiences.
Specifically, OMRE conducts market and user-centered research and evaluation in order to:
Identify perceived cancer-related needs of diverse audiences
Inform the design and development of NCI resources and ensure that they are appropriate and effective, and reach the intended audiences
Monitor audience trends
Build on and advance the Institute's knowledge base and positioning in the field of user-centered informatics research
Assess the impact of resources and activities
The research conducted in OMRE helps ensure that NCI communication and education resources are appropriate, useful, and effective. This NCI office is requesting OMB to review this package to request a reinstatement with change on OCE’s package entitled, “Pretesting, Formative Research, and Customer Satisfaction of NCI’s Communication and Education Resources,” OMB#0925-0046, expiration: 2/28/13. The only planned change is a decrease in burden requested. Purpose, use, methodology and design remain the same as the previously approved submission.
The National Cancer Institute (NCI), established under the National Cancer Act of 1937, is the Federal Government's principal agency for research on cancer cause, prevention, detection, diagnosis, treatment, and rehabilitation, and for the dissemination of information for the control of cancer. Current authorization for NCI's education and information dissemination activities is contained in Section 410 of the Public Health Service Act (42 USC § 285).
NCI’s OCE is the main office within the NCI that provides communications expertise within the Institute and between NCI and a variety of organizations and audiences, including Congress, other executive agencies, state and local governments, scientific and medical communities and institutions, voluntary groups, the press, the general public, and cancer patients. The OCE, which supports NCI’s priorities through activities that span NCI programs, specializes in the design, implementation, and evaluation of education programs over the entire cancer continuum, including prevention, screening, diagnosis, treatment, survivorship, and palliative care. The office also manages NCI initiatives that address specific challenges in cancer research and treatment. Health care providers, professional societies, patient groups, federal agencies, and the public are audiences for OCE’s educational programs and products/materials.
Within OCE, the Office of Market Research and Evaluation (OMRE) ensures that the National Cancer Institute’s (NCI) communication and education resources are appropriate, useful, and effective. OMRE uses scientific methods to: identify the cancer-related needs of diverse audiences; inform the design and development of NCI resources; monitor audience trends; and, assess the impact of resources and activities.
Information programs and campaigns within NCI create and use a variety of media including print (e.g., brochures, posters, fact sheets, information kits), broadcast (e.g., public service announcements, video news releases), and electronic formats (e.g., Internet, listservs), as well as direct response (Cancer Information Service) to inform and educate the public and health professionals about cancer. Production of these materials is the major way that the Institute relays messages to the audiences it is mandated to reach.
It is OCE’s responsibility to ensure that the materials produced by NCI can be understood and are well-received by intended audiences (Formative Research/Pretesting), and meet the satisfaction of NCI’s audiences (Customer Satisfaction). Formative research/pretesting helps ensure that messages have the potential to be received, understood, and accepted by those for whom they are intended; while the customer satisfaction research helps NCI ensure the relevance, utility, and appropriateness of the many educational programs and products that the Office produces. See Attachment 1 (Explanation of Types of Research) for additional details on the purpose of and distinctions between the two types of research. Also, Attachment 2 contains a listing of studies approved along with information collection methods used OMB 0925-0046 from 2010 to current.
OCE is requesting approval of this generic clearance to continue conducting formative research, pretesting, and customer satisfaction activities through a reinstatement with change request to OMB of No. 0925-0046, expiration date 2/28/13.
A.2. Purpose and Use of the Information
Formative research, pretesting, and the assessment of customer satisfaction are integral parts of NCI’s overall plan for its communications strategies. Pretesting of print, broadcast, and web products and information services assesses the potential effectiveness of these products while they are still in the developmental stage. Given the large number and wide spectrum of cancer education programs and products OCE and NCI develop and disseminate, NCI requires accurate, timely, and useful information about the relevance, usefulness, and appropriateness of these products to its customers.
The formative research process is used to determine whether or not a draft message or message concept is effective in reaching and communicating with its audience. Pretesting involves presentation of draft messages designed to convey specific information to a sample of the audience for whom the materials are intended. These respondents are asked to give their reaction to the messages through either individual or group interviews. Information collected to determine the level of customer satisfaction with products helps NCI identify strategies for improving the accessibility of materials/programs, their user-friendliness, and their relevance to the needs of cancer patients and their families, health educators and interventionists, cancer advocates, cancer information specialists, and health care professionals. Research explores various domains shown in the table below.
Factors assessed |
Research questions |
Attention |
Do the messages attract and/or hold the audience's attention? |
Comprehension |
Are the messages and language clearly understood, and does the main theme of the message resonate with the audience? |
Personal relevance and Self-efficacy |
Do members of the target audience perceive the message as personally relevant, and do they see themselves as capable of acting on the message provided? |
Believability |
Is the message and/or its source perceived as credible? |
Acceptability |
Is there anything in the message that may be offensive or unacceptable to the target audience, and is the piece culturally sensitive? |
Accessibility |
Will members of the target audience be able to find the information or message with relative ease, considering factors such as the availability of products in multiple formats (e.g., print materials, videos, online documents, CDs, audio tapes) and in a variety of settings (e.g., clinics, hospitals, doctors’ offices, community organizations, libraries)? |
Usability |
How likely is the respondent to use the information in the format provided, and is the content provided in a logical, organized and user-friendly way? And, do the topics addressed by NCI and OCE products meet the needs, concerns and interests of its customers? |
Behavioral intent |
Do respondents think they will take action as a result of seeing/hearing the message? |
Readability and clarity |
Can the reader understand publications and educational materials, both in terms of the needs of low-literacy audiences and with respect to plain language principles and design? |
Availability |
Are NCI and OCE products present for low-English fluency audiences? |
Cultural appropriateness |
Are foreign-language translations or adaptations of products accurate and available for customers? |
Respondents' input and reactions to each of these areas provide insight into how the audiences for these messages may react, how the messages should be formulated or revised to communicate most effectively, and the usefulness, relevance and appropriateness of these products to its customers.
Other information that may be gathered on respondents regarding gender, age, socioeconomic level, race/ethnicity, and family medical history provides a basis for evaluating whether the messages may be perceived differently by different segments of the audience. For example, selected age groups may find a particular brochure or message on cancer prevention more relevant than other age groups.
Systematic formative research and pretesting has been widely adopted by health education program planners as an integral step in the development and targeted dissemination of messages and materials. Through this research, NCI is able to:
Understand characteristics of the target audience—its attitudes, beliefs, and behaviors—and use these in the development of effective communications tools;
Design messages and select formats that have the greatest potential to influence the target audience’s attitudes and behavior in a favorable way;
Determine the best promotion and distribution channels to reach the target audience with appropriate messages;
Expend limited program resource dollars wisely and effectively; and
Results of past pretesting and customer satisfaction efforts have been instrumental in helping NCI carry out its legislative mandate. This generic has hosted a wide variety of sub-studies from formative research, pretesting messages/materials/strategies/formats, as well as customer satisfaction efforts (Attachment 2).
A.3. Use of Information Technology and Burden Reduction
Information collection may be conducted using a variety of methodologies and technologies, such as one-on-one interviews, group interviews, or self administered questionnaires, depending upon the target audience being questioned and the subject matter being addressed.
As computer technology has continued to improve and become more widespread, opportunities to pretest messages on the Internet using either Web site questionnaires or on-line focus groups with Internet users have increased. Improved technology in the collection and processing of data has the potential to reduce the time burden for respondents and data collectors. For example, respondents can access and respond to data collection requests at a time and place that is convenient to them, eliminating the need to travel for in-person or group interviews. Wherever possible, NCI will make use of Web- or computer-based data collection methods. Transmission of data collection instruments and responses by electronic mail or facsimile will be utilized as appropriate (for example, with intermediary audiences such as NCI’s Cancer Information Service or Public Affairs Network). NCI anticipates that of the majority of data will be collected electronically.
Possible information technologies to assess customer satisfaction include
computer-assisted telephone interviewing (CATI), computer-assisted personal interviewing (CAPI), audio and computer-assisted self-interviewing (ACASI), on-line surveys, and focus groups. Each technology/method is briefly described below:
CATI’s technological capabilities include automated dialing, scheduling unanswered calls or interrupted interviews for efficient callbacks, random selection of respondents, automated skip patterns, instantaneous out-of-range checks, insertion of information from one question to guide a subsequent question, and the automated generation of databases for subsequent analysis. CATI will be utilized when geographic diversity is important and participants come from hard-to-recruit populations, such as physicians or Native Americans. CAPI technology allows interviewers to ask questions of a respondent using a computer to enter data. ACASI software technology offers many advantages of CAPI technology, but removes the need to have a person administer an interview. Instead, survey questions are pre-recorded and played back through the sound system of a computer, which the respondent can listen to privately by using headphones. Respondents select an answer by pressing a key that corresponds to one choice shown on the screen, after which answers are fed directly into a computer database.
ACASI surveys can also be administered over a telephone by entering the response on the telephone keypad. ACASI technology is particularly useful in administering surveys to low-literacy populations or when addressing sensitive topics that respondents may not feel comfortable discussing in the presence of someone else.
On-line surveys represent an especially convenient option for eliciting feedback from consumers of Web-based products. Respondents complete online surveys regarding a product and then submit the data electronically over the Internet. With online surveys, respondents can easily submit feedback during or immediately after using a Web-based product. They also allow participation from international audiences with virtually no additional costs.
Technology now enables the conduct of focus groups, which traditionally take place in-person with a moderator facilitating a discussion regarding a product, issue, or program with a small group of individuals in a designated physical location. Depending on factors such as geographic distribution and schedules of NCI customers and on the nature of the products and services under investigation, focus groups, when appropriate, may be implemented using a variety of technology-based formats:
Videoconferencing uses video and satellite technology to allow a group of participants located in multiple geographic locations to interact with one another both visually and aurally. A facilitator and a technical team located in a hub site maintain the video and audio connections among participating sites.
Internet or online conferencing is especially useful for discussions with specific individuals or international participants. This format functions as a sort of “chat room” in which a moderator intercepts and distributes e-mail transmissions from participants who have logged onto a specially designated Web site.
Teleconferencing uses telephone technology to facilitate an exchange among participants located in multiple geographic locations. Participants dial into a specially designated phone number or “bridge line” that is moderated by a focus group facilitator.
Advanced Strategies Lab (ASL) is a qualitative research process that uses an online interactive discussion. In the 90-minute, online ASL sessions, participants receive instructions and are asked questions from the moderator over the phone on a conference call. Participants’ answers are typed into their designated dialogue box (which hides their identity from the other participants to ensure confidentiality). The facilitator then guides the group through online brainstorming, discussion, and assessment exercises. The ASL’s software automatically formats and tabulates data as the session proceeds. Summaries of survey results are available seconds after respondents complete a question, and full verbatim reports are available within hours after a session.
A Privacy Impact Assessment (PIA) is has been conducted and approved by HHS. The name of the IT system is “NIH NCI OCE Office of Market Research and Evaluation Surveys” (Attachment 3).
A.4. Efforts to Identify Duplication and Use of Similar Information
The areas in which information needs to be gathered (as described in A.2. above - attention, comprehension, etc.) to pretest effective cancer messages (brochures, PSAs, media campaigns, website, etc.) are generally similar from pretest to pretest. However, since the cancer field is so diverse and complex, and each message is essentially different, new data collection instruments generally must be prepared for each pretest. As each new message, strategy, or product is developed, NCI reviews existing literature and data bases, including pretesting reports on existing messages and materials, and consults with outside experts to evaluate available information on similar messages with comparable audiences.
NCI will continue to assess other NIH active generic ICs upon submission of applications to conduct each sub-study. Currently, the existing NIH active generic ICs are very site- or method-specific. Though other Institutes at NIH have a Generic submission, none would be able to accommodate the population and content that the National Cancer Institute’s Generic IC anticipate needing in the areas of customer satisfaction, pretesting NCI communication messages, and formative research. Additionally, NCI has an internal review process for surveys that will be used by this generic to assess the quality of each survey prior to its use. The NCI will provide direct oversight for any and all surveys conducted under this generic clearance to avoid duplication of effort and information collected.
A.5. Impact on Small Businesses or Other Small Entities
Small businesses that are non-profits and independently-owned may be participants in this generic submission. The small businesses we may include are physicians, health care providers, and highly specialized individuals may sometimes be the target audience for NCI’s communication information and customer satisfaction materials. When small businesses are asked to complete an information collection, all efforts will be made to reduce their burden by using a short questionnaire/survey and interviewing fewer small businesses than larger ones.
A.6. Consequence of Collecting the Information Less Frequently
For the most part, formative research, pretesting, and customer satisfaction information will be collected only one time for each print, broadcast, or electronic message, product/material, or strategy tested. However, there may be occasion where a pre- and post-test to assess differences in communication and/or satisfaction messages may be useful for a particular sub-study. Additionally, previous respondents may be contacted to participate in follow-up studies if they have originally granted consent for such and if the subsequent study uses that population.
A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
NCI recognizes the need to collect information in a manner that places minimal burden on each respondent. Therefore, NCI anticipates that information collections under this clearance package will comply with 5 CFR 1320.5(d)(2) requirements with only two anticipated exceptions.
When NCI requires responses to a self-administered written questionnaire in less than 30 days, receipt of the questionnaire is generally preceded by advance notification to respondents explaining the purpose of the questionnaire, the approximate length of time that the questionnaire will take, and the voluntary nature of participation. All efforts are made to keep such questionnaires short and focused.
Because NCI’s pretesting activities are often qualitative in nature, the results are not generalizable to the population at large or to the particular target audience under study. However, the nature of pretesting is such that generalizability is not a critical feature; the emphasis is on obtaining timely, useful information that can be fed back into the development of new messages or materials or the revision of existing ones.
There are no other special circumstances.
A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside
Agency
The 60-Day Federal Register notice soliciting comments on this study prior to initial submission to OMB was published on January 2, 2013 (Volume 78, Page 105). Two public comments were received and responded to. One by JeanPublic and the other by a research company asking about marketing.
NCI, along with other Public Health Service agencies, has been a leader in the development of methods for developing, testing, and disseminating health information. The work of many advisers over many years has brought us to where we are today. A number of outside health communications experts were consulted to review the plans contained herein for program development research and evaluation of NCI communications programs and their comments and suggestions have been incorporated into these data collection plans.
Although all consumer materials, as well as materials produced for the media, must go through DHHS approval procedures, NCI is not required to coordinate with other Federal agencies in its education, information dissemination, and evaluation activities. Nevertheless, NCI has consulted with other agencies in the past and will continue to do so in the future, as appropriate, to help ensure accuracy and consistency, and to avoid duplication of effort. It should also be noted that many of the pretesting and formative research efforts conducted by NCI have been requested and used by outside agencies to inform their own communications activities. OCE also consults with several internal experts across NCI, most frequently in NCI’s Division of Cancer Control and Population Sciences (DCCPS) on the conduct of study design, methods, and data analysis and reporting. Any collaborations and consultations that occur for specific sub-studies will be mentioned as part of the request for approval in the sub-study.
A.9. Explanation of Any Payment or Gift to Respondents
It is possible that some information collection activities will entail small payments or gifts to respondents. Small amounts of money, a free meal or snack scheduled around the time of the pretest, and/or renumeration for parking and/or transportation are most often used, particularly when recruiting hard-to-reach and minority respondents. Renumeration may also be necessary in some sub-studies where highly specialized individuals are invited to participate in a survey, and will cover the cost of transportation and other types of local expenses.
Research has shown the advantages of providing a small incentive for improving response rates and decreasing item nonresponse, especially in mail and telephone surveys.1 Studies of participants in the original National Health and Nutrition Examination Surveys (NHANES) found that response rates for those told they would receive remuneration versus not were 82% and 70%, respectively.2 The National Survey of Family Growth conducted an experiment with remuneration of respondents and found that incentives increased response rates, reduced interviewer labor (broken appointments and callbacks), and improved data quality.3 In the 1999 Observing Protein and Energy Nutrition study, remuneration was credited as contributing to high response and retention rates.4
Instances for offering an incentive will be determined on a case-by-case basis (depending on the particular information collection design). The following are the kinds of situations for which respondents may be paid or given a gift:
Physicians who participate in a survey on their opinions about the relevance of materials for their patients may receive a small payment or donation to a charitable organization in their honor (for example, $100).
Individuals who participate in in-person focus groups may receive an honorarium (perhaps $50) to cover their time, transportation costs, and childcare expenses.
Health educators who support the acquisition of data related to customer satisfaction with specific educational programs and products may be able to request and receive certain quantities of materials that exceed the limits usually established for those materials.
Circumstances, however, do not always require that renumeration be given; many audiences including the public, patients, survivors, and other health professionals often participate gratis because of their interest or involvement in the topic, or as a professional courtesy. For example, in situations when the general public is completing an online survey, no remuneration will be involved unless influenced by other factors.
A.10. Assurance of Confidentiality Provided to Respondents
Information provided by respondents will be kept confidential and private, except as otherwise required by law. This will be communicated to respondents by means of introductory letters, explanatory texts on the cover pages of questionnaires, scripts read prior to focus groups or telephone interviews, and consent forms. Respondents will also be advised of the following: the nature of the activity; the purpose and use of the data collected; NCI sponsorship; and the fact that participation is voluntary at all times. Because responses are voluntary, respondents will be assured that there will be no penalties if they decide not to respond, either to the information collection as a whole or to any particular questions.
As a further guarantee of confidentiality, all presentation of data in reports will be in aggregate form, with no links to individuals preserved. Reports will be used only for research purposes and for the development of communication messages and educational materials. Only NCI staff and contractor personnel conducting the information collection will have access to individual-level survey, interview, or focus group data. All project/contractor staff conducting the information collection will sign a confidentiality agreement, and all electronic and hard-copy data will be maintained securely throughout the information collection and data processing phases. While under review, electronic data will be stored in locked files on secured computers; hard-copy data will be maintained in secure building facilities in locked filing cabinets. Before any data are released for public use data sets, any identifying information will be stripped from each respondent’s record and the identifying information will be destroyed.
The NIH Privacy Act Officer has reviewed the work scope of this proposal to determine whether the Privacy Act is applicable to this data collection and will be asked to review the protocol of each collection under this blanket clearance to ensure that NCI adheres to privacy requirements (see Attachment 4, Memo from NIH Privacy Act Officer).
Although some personally identifiable information will be collected, data will not be retrieved by personal identifiers unless the respondent voluntarily agrees to provide the information, so he/she can be contacted for follow-up (in rare situations). Instances could arise for activities that, for example, gather and retain respondent names and contact information (such as addresses, telephone numbers, or email addresses). This information would be used to measure customer satisfaction information regarding an NCI/OCE product at one time and then at some point later (for example, to learn whether customers have gone back to the product for additional review or information.)
Before submitting each sub-study for OMB review, NCI always submits related IRB paperwork to the Office of Human Subjects Research Protection (OHSRP). However, pretesting efforts described in this proposal are typically considered exempt from the “Regulations for the Protection of Human Subjects,” in accordance with paragraph (b)(3) of 45 CFR Sec. 46.101
(http://citfm.cit.nih.gov/ohsr/nih/guidelines/GrayBooklet82404.pdf).5 OCE understands and has experience with the process and plans to continue submitting for review/exemption, using Exemption Request forms (Attachment 5) found here: http://ohsr.od.nih.gov/ (Note: Please contact Nina Goodman with questions).
A.11. Justification for Sensitive Questions
As mentioned in sections A.2. and A.10. above, some studies require the inclusion of people who match selected characteristics of the target audience that NCI is trying to reach. Therefore, personally identifiable information (PII) such as race/ethnicity, income, education and/or medical/health status, are required to be asked on the initial screening questionnaire used for recruiting. Potential participants are informed that this is being done to make sure that NCI speaks with the kinds of people for whom its messages are intended. Again, respondents are assured that the information is voluntary and will be treated as confidential. All information on race/ethnicity will comply fully with the standards of OMB Statistical Policy Directive No. 15, October 1997. (http://www.whitehouse.gov/omb/fedreg/1997standards.html).
Since NCI communications are concerned with the detection, diagnosis, treatment, and prevention of cancer, some projects may involve asking questions about (or discussing) how one perceives his/her own personal risk for serious illness. Fears of cancer and experiences with cancer may also be covered. This information is needed to gain a better understanding of the target audience so that the messages, strategies, and materials designed will be appropriate and sensitive. Questions of this nature, while not as personal as those on sexual behavior or religious beliefs, still require some sensitivity in how they are worded and approached. In face-to-face data collections, questions of this kind are generally asked later in the interview or group discussion when respondents are more comfortable with the interview situation and are more at ease with the interviewer/moderator. As noted in section A.10., participants are informed in advance about the nature of the activity and the voluntary nature of their participation. The interviewer/moderator makes it clear that they do not have to respond to any question that makes them uncomfortable, and that no questions of a sensitive nature will be asked in the course of collecting information from respondents.
Raw data from data collections that include sensitive information (for example, screening questionnaires and audio tapes) are not retained once the data have been extracted and aggregated; nor does the information become part of a system of record containing permanent personally identifiable information that can be used for retrieval.
A.12. Estimates of Annualized Burden Hours and Costs
The types of respondents may include, but are not limited to the general public (patients, survivors, family members), physicians, other health professionals, and researchers. The Category of Respondents will primarily be Individuals and Households, but at times may include Local, State, or Federal Government and the Private Sector. The number of respondents to be included in each sub-study will vary, depending on the target audience and the nature of the message/materials/program being assessed. Typically, the frequency of responses per respondents would be once. However on occasion, respondents may give permission for a follow-up survey or interview and since this happens fairly infrequently it is not being accounted for in Table A.12-1. Sub-study submissions will indicate when NCI anticipates follow-up surveys/interviews with a given number of respondents. The average time, or burden per respondent, can range from five minutes to 90 minutes, with the average being around 12 minutes. It is also difficult to pinpoint the actual number of studies required during the next three years; we can only anticipate need based on past studies. Annual burden, therefore, is one-third of the total figures presented here. It is estimated that on an annual basis, the total number of respondents is 33,000 and the annual estimated burden is 6,600 hours.
Table A.12-1 Estimate for Burden Hours For Three Years (Generic Study)
Category of Respondents |
Form Name |
Number of Respondents |
Frequency of Response per Respondent |
Time Per Response (in hours) |
Burden Hours |
Individuals, Households, Local, State, and Federal Governments, and Private Sector |
Focus Groups, Individual In-Depth Interviews, Brief Interviews, Surveys, Website Usability Testing |
33,000 |
1 |
12 |
6,600 |
Totals |
|
33,000 |
|
|
6,600 |
Table A.12-2 presents the approximate cost to respondents over the 3-year life of the project. Annual cost, therefore, is one-third of the total figures shown. The mean hourly wage rate to members of the general public is $21.74 per hour. While physicians (general practitioners) sometimes participate gratis in telephone or self-administered surveys (time permitting), the mean hourly wage rate is substantially higher at $89.00 per hour (http://www.bls.gov/oes/current/oes_nat.htm)6. For the most part, non-physician health professionals and researchers have a mean hourly wage rate in between the two. For Table A.12-2, the highest wage rate is used to calculate the total cost.
Additionally, the costs assume that the general public will only be completing focus groups, and the physicians completing individual in-depth interviews, etc. however, this very well may not be the case. It is difficult to estimate the number of each type of respondents that will complete different survey methods and for the ease of calculation, it was calculated as such. Therefore, the costs may increase or decrease, based on this assumption.
A.12-2 Cost to Respondents For Three Years (Generic Study)
Type of Respondents |
Total Burden Hours |
Hourly Respondent Rate |
Total Cost |
Individuals, Households, Local, State, and Federal Governments, and Private Sector |
6600 |
$20-100 |
$660,000 |
A.13. Estimate of Other Total Annual Cost Burden to Respondents and Recordkeepers
There are no capital or start-up costs to the data collection efforts requested; nor are there any costs associated with operation, maintenance or purchase of services.
A.14. Annualized Cost to the Federal Government
The estimated annualized cost to the federal government is approximately $92,740, which amounts to a total estimated cost of $278,220 for a period of three years. Table A.14-1 shows the breakdown of estimated costs, with figures that include the costs of study design, facility rental (e.g., for focus groups), data collection, analysis, and report/publication writing. This estimate also includes monitoring by the Government Project Officer and involvement by NCI's Senior Analyst and supporting staff, projected to be about 500 hours of effort a year. Given a total NCI personnel average cost of $60.00 per hour, $30,000 would be spent annually on Government staff salaries (or $90,000 over the 3-year period).
Table A.14-1 Annualized Costs to the Federal Government
Annual costs for NCI staff to plan, conduct, and analyze the outcomes of information collection activities:
|
Managerial |
FTE |
$3,000 |
Professional |
FTE |
$25,000 |
|
Support |
FTE |
$2,000 |
|
IN-DEPTH INTERVIEWS
|
$55,440 |
||
FOCUS GROUPS
|
$105,840 |
||
INTERVIEWS - approximately 3 central location interview studies at $11,340 each - approximately 10 telephone interview studies at $3,780 each - approximately 4 gatekeeper reviews (interviews) at $11,340 each
|
$34,020 |
||
SURVEYS - approximately 4 self-administered questionnaire studies at $5,670 each - approximately 2 telephone omnibus surveys at $15,120 each
|
$22,680 $30,240 |
||
TOTAL |
$278,220 |
||
A.15. Explanation for Program Changes or Adjustments
This information collection request is considered a reinstatement with change. This Generic submission has been very active over the past couple of years and anticipates a continued activity at the current level. Since less than half of the previously requested burden hours was used over the course of the past three years, the program is asking for a slight reduction in burden hours. Additionally, with the approval of the NCI Fast Track submission (OMB No. 0925-0642), it is anticipated that some of the submissions that would have been a sub-study under this Generic will now seek approval under the Fast Track. Another factor that should be considered is a reduction in the budget, thus reducing the number of anticipated sub-studies over the next three years.
A.16. Plans for Tabulation and Publication and Project Time Schedule
Evaluation/research staff will review the material to be tested; discuss the objectives with the individuals responsible for development; determine the analytic questions to be addressed; and then prepare the research procedures, instruments, and data analysis plan. The analysis conducted for each study will be determined by the objectives of the research, the messages being tested, and the audience for the messages. Specifics of each analysis cannot be determined until the messages to be tested are prepared. Research techniques may include qualitative analysis, customer satisfaction analysis, descriptive statistics, statistical analysis and parametric statistical tests. Analysis may be conducted using cross-tabulation procedures, with categorical variables; or between-group procedures, with continuous variables.
While the primary purpose of all studies is to provide information to the developers of the messages/materials/services for the purpose of improving them, NCI shares information internally and also makes results available to a variety of health program planners at Government agencies, voluntary organizations, health professional organizations, and medical institutions. Information collected will be compiled and presented in reports and briefings for staff from OCE and other divisions within NCI and NIH. Reports will include information regarding respondent demographics, basic descriptive data with OCE products and services, comparisons across demographic and customer subgroups, recommendations for improving programs and products, and analyses of longitudinal changes. In addition, NCI may also publish results in journals and present the findings of its research at meetings of professional associations, for example, the American Public Health Association and the Society for Public Health Education. Formative research conducted by OCE is also sometimes summarized in news-related publications such as the NIH Record.
The specific messages and materials/service that will be evaluated and the timing of these studies are not known at this time. While the research period varies somewhat depending on the complexity of the testing and number of respondents required, the typical study will require approximately 12 weeks from initial design to preparation of the report of pretest findings. A schedule for a typical pretest is shown below:
A.16-1 Project and Publication Timeline
Activity Time Schedule
Initial review of research questions 1-2 weeks after OMB approval
Write data collection instrument 3-4 weeks after OMB approval
Preparation of design and plan 4-5 weeks after OMB approval
Review of design 6-7 weeks after OMB approval
Collection of data 7-8 weeks after OMB approval
Analysis of data 9-10 weeks after OMB approval
Write report of findings 11-12 weeks after OMB approval
Develop manuscript (when seeking publication) 4-6 months after OMB approval
Submit for publication 7-8 months after OMB approval
A.17. Reason(s) Display of OMB Expiration Date is Inappropriate
NCI will continue displaying the OMB control number and expiration date in the upper right-hand corner of all data collection instruments.
A.18. Exceptions to Certification for Paperwork Reduction Act Submissions
NCI is in full compliance with the provisions contained within the Certification for Paperwork Reduction Act.
1 E. Singer, J. Van Hoewyk, and M. P. Maher, “Experiments with Incentives in Telephone Surveys,” Public Opinion Quarterly, Vol. 64, No. 2, Summer 2000, pp. 171-188; A. H. Church, “Estimating the Effect of Incentives on Mail Survey Response Rates: A Meta-Analysis,” Public Opinion Quarterly, Vol. 57, No. 1, Spring 1993, pp. 62-79.
2 U.S. Department of Health, Education, and Welfare, “A Study of the Effect of Remuneration Upon Response in the Health and Nutrition Examination Survey, United States,” Vital and Health Statistics, Series 2-No.6, 1975.
3 W. D. Mosher, W. F. Pratt, and A. P. Duffer, “CAPI, Event Histories and Incentives in the NSFG Cycle 5 Pretest,” American Statistical Association, 1994 Proceedings of the Section on Survey Research Methods, Vol. 1, 1995, pp. 59-63.
4 A. F. Subar, V. Kipnis, R. P. Troiano, et al., “Using Intake Biomarkers to Evaluate the Extent of Dietary Misreporting in a Large Sample of Adults: the OPEN Study.” American Journal of Epidemiology, Vol. 158, 2003, pp. 1–13.
5 Reference pages 8 and 21 for details about exemptions.
6 This is from the May 2011 National Occupational Employment and Wage Estimates, United States.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | McCarthyK |
| File Modified | 0000-00-00 |
| File Created | 2021-01-29 |