Supporting Statement B For:
State and Community Tobacco Control (SCTC) Research
Initiative Evaluation (NCI)
September 2, 2014
Elizabeth M. Ginexi, Ph.D.
Tobacco Control Research Branch
Behavioral Research Program
Division of Cancer Control and Population Sciences
National Cancer Institute
9609 Medical Center Drive, Room 3E564, MSC 9761
Bethesda, MD 20892-9761
Telephone: 240-276-6765
Fax: 240-276-7907
E-mail: [email protected]
Table of Contents
B. Statistical Methods 1
B.1 Respondent Universe and Sampling Methods 1
B.2. Procedures for the Collection of Information 3
B.3 Methods to Maximize Response Rates and Deal with Nonresponse 7
B.4 Test of Procedures or Methods to be Undertaken 9
B.5 Individuals Consulted on
Statistical Aspects and Individuals
Collecting and/or
Analyzing Data 10
Attachments
1. SCTC Scientists Web Survey
2. Affiliated Partners Web Survey
3. PI/Co-PI Telephone Interview Guide
4. Affiliated Partner Telephone Interview Guide
5. Pilot Project Telephone Interview Guide
6. Working Group Telephone Interview Guide
7. Coordinating Center Telephone Interview Guide
8. PI/Co-PI Expert Panel Moderator Guide
9. Expert Panel Informed consent
10. Privacy Impact Assessment
11. Individuals Consulted in Development of Conceptual Model
12. Privacy Act Memo
13. IRB Approval Letters
14. Email Announcement
15. Email Invitation Letter – Web Survey - SCTC Scientists
16. Email Invitation Letter – Web Survey - Affiliated Partners
17. Email Invitation Letter - Telephone interview - SCTC Scientists
18. Email Invitation Letter - Telephone interview - Affiliated Partners
19. Email Invitation Letter - Expert Panel
20. Email Reminder- Survey - SCTC Scientists
21. Email Reminder- Web Survey - Affiliated Partners
22. Email Reminder- Telephone interview - SCTC Scientists
23. Email Reminder- Telephone interview - Affiliated Partners
24. Telephone Follow-up Reminder- Web Survey
25. Telephone Follow-up Reminder - Telephone Interview
26. Email Follow-up Reminder to Schedule Telephone Interview
27. Telephone Script to Schedule Telephone Interview
28. Email Confirmation for telephone Interview
The State and Community Tobacco Control Research Initiative (SCTC) is a program that targets four high priority tobacco control research areas at the state and community level: (1) Secondhand smoke policies; (2) Tobacco tax and pricing policies; (3) Mass media countermeasures and community and social norms; and (4) Tobacco industry practices. The NCI is undertaking a process evaluation study to assess the dissemination, implementation, and community collaboration processes of the SCTC grantees and their respective state and community partners and stakeholders.
We are proposing to use statistical methods to obtain the outcome data measures from our study population of Principal Investigators (PIs), Co-Principal Investigators (Co-PIs), affiliated partners and stakeholders. There are three modes of data collection: web-based surveys (Attachments 1 and 2), telephone interviews (Attachments 3 through 7) and expert panels (Attachment 8). Statistical methods for the web-based survey will consist of descriptive statistics (e.g. percentages, means, medians, standard deviations, as appropriate), cross-tabulations, and graphical summaries. The web-based survey analyses will be performed in SPSS for descriptive statistics, and in UCINET and NodeXL for social network analysis. The telephone interview and expert panel analyses will be performed using Excel.
The target populations for this process evaluation are the SCTC Research Initiative science network comprised of the PIs and Co-PIs from the seven SCTC research centers, their affiliated tobacco control partners, and NCI representatives. The seven research centers are located at Emory University; University of Illinois at Chicago, Institute for Health Research and Policy; Mayo Clinic Arizona; University of North Carolina Chapel Hill; University of California San Diego; the University of California San Francisco; and the SCTC Coordination Center located at Research Triangle Institute (RTI). The affiliated tobacco control partners are comprised of 3-5 individuals nominated by each PI at the research centers noted above. Affiliated partners refer to SCTC collaborators who are not SCTC grantees and are based outside of funded research centers. Affiliated partners can include representatives at state or local tobacco control programs, public health and community based organizations, Federal or State agencies, organizations hired to consult on SCTC projects or workgroups, as well as other academic institutions.
The target population for the web-based survey (Attachments 1 and 2) will include all PIs, Co-PIs and affiliated partners for a total of 131 potential participants. A total of 60 PIs and Co-PIs and 71 affiliated partners will be invited to participate in the survey with an expected number of 57 PI and Co-PI participants and 50 affiliated partner participants. The response rates for the survey are expected to be at about 95% for PIs and Co-PIs and 70% for affiliated partners.
The target population for the telephone interviews (Attachments 3 through 7) will include PIs, Co-PIs, and affiliated partners for a total of 131, comprised of 60 PIs and Co-PIs, 71 affiliated partners and two NCI representatives. Of the 60 PIs and Co-PIs, 21 will be sampled to participate as core project leads and 14 as component leads (6 from Pilot Projects, 6 from Working Groups, and two RTI Coordinating Center representatives). Additionally, of the 71 affiliated partners, 21 will be sampled to participate, and two NCI representatives will be selected to participate. The number of participants is expected to total 56. A strategic sample will be drawn; balanced by geographical area, organizational type, and project focus.
The target population for the expert panel (Attachment 8) will include all PIs and Co-PIs for a total of 60 potential participants. Two expert panel sessions will be held comprised of nine PIs and Co-PIs in each session. The number of participants is expected to total up to 18. The expert panel participants will be balanced by geographical area, organizational type, and project focus. No more than nine PIs and Co-PIs may participate in all three data collections modes: the survey, telephone interview, and expert panel. Though different factors may be responsible for non-response or non-cooperation, we anticipate that this will be due to (1) individuals may be too busy to participate, and in rare cases (2) potential lack of familiarity with on-line surveys for some individuals. The number of potential and expected participants by type of data collection is presented in Table B.1-1.
Table B.1-1 Number of Potential and Expected Participants by Data Collection Type
Data Collection Type |
Number of Potential Participants |
Number of Expected Participants |
Web Survey |
131 |
107 |
Telephone Interview |
131 |
56 |
Expert Panel |
60 |
18 |
Data collection procedures for the web-based survey, the telephone interview and the expert panel are outlined below. Within three weeks of OMB approval, all PIs, Co-PIs and Affiliated Partners with a valid email address will be sent an NCI email study announcement about the SCTC Process Evaluation (Attachment 14). Please note that while as described below some non-responders may be contacted several times in the course of the study, the majority will be contacted only once or twice for each survey or interview administration. Given study participant involvement in the SCTC initiative we expect response rates throughout the evaluation to be fairly high and thus the need to re-contact individuals – especially the PIs and Co-PIs – to be minimal.
Data collection procedures for the survey are described below and presented in Table B.2-1:
Two weeks after the NCI email announcement, potential participants for the web-based survey will be sent an email invitation to participate. The invitation letter requests the individual’s participation, introduces the purpose of the survey, includes instructions on how to participate, and provides contact information in case of queries (Attachment 15-16).
One week after the email invitation, a reminder email letter will be sent only to non-responders to encourage participation in the study. The reminder email requests the individual’s participation, and provides contact information in case of queries (Attachment 20-21).
One week after the first reminder email, a second reminder email letter will be sent only to remaining non-responders to encourage participation in the study. The reminder email requests the individual’s participation, and provides contact information in case of queries (Attachment 20-21).
One week after the second reminder email, telephone follow-up will be conducted with only the remaining non-responders for whom there is a telephone number available (Attachment 24). If there is no telephone number available, a staff member will investigate (via Google and other databases) the correct contact information. Section B.3.1 presents the study methods to address non-response and maximize response rates.
Table B.2-1 Procedures for Survey
Timeline
|
Activity |
3 weeks after OMB approval |
NCI email study announcement sent to all potential participants |
2 weeks later |
Email invitation letter sent to all potential participants for survey |
1 week later |
Reminder email notice sent only to survey non-respondents |
1 week later |
Second reminder email sent only to non-responders |
1 week later |
Telephone Follow-up only with non-responders with telephone number available. Non-responders with no telephone number will be traced. |
Data collection for the telephone interviews will begin once the survey data collection has been completed in order to minimize respondent burden. As described above, reminder and follow-up contacts occur only with non-responders. Assuming high response rate among PIs and Co-PIs, we expect reminder and follow-up contacts to be minimal. Data collection procedures for the telephone interview are described below and presented in Table B.2-2:
Ten weeks after the NCI email announcement, potential participants for the telephone interviews will be sent an email invitation to participate. The invitation letter requests the individual’s participation, introduces the purpose of the evaluation, requests an email reply with suggestions of 2-3 convenient times to be contacted for the interview, and provides contact information in case of queries (Attachment 17-18).
One week after the email invitation, a reminder email letter will be sent only to non-responders to encourage participation in the study. The reminder email provides the same information as the invitation email (Attachment 22-23).
One week after the reminder email, telephone follow-up will be conducted only with non-responders for whom there is a telephone number available (Attachment 25). If there is no telephone number available, a second reminder email will be sent (Attachment 22-23). If there is no response received within two weeks from either the telephone follow-up or the second reminder email, the potential participant will be considered a non-respondent and removed from the sample. A replacement will be selected from the strategic sample group. Section B.3.1 presents the study methods to address non-response and non-cooperation.
When a potential participant replies with general interest in participating, a follow-up email will be sent stating that someone will call them to schedule the interview (Attachment 26). If a potential participant does not reply to schedule the interview, the scheduler will call the individual to schedule the interview using a telephone script to schedule the interview (Attachment 27).
When a potential participant accepts the invitation to participate in an interview, an email confirmation will be sent within two business days confirming the interview date and time and the best telephone number at which to reach the interviewee (Attachment 28).
Table B.2-2 Procedures for Telephone Interview
Timeline
|
Activity |
3 weeks after OMB approval |
NCI email study announcement sent to all potential participants |
10 weeks later |
Email invitation letter sent to all potential participants for telephone interview |
1 week later |
Reminder email notice sent only to non-respondents |
1 week later |
Telephone Follow-up only with non-responders with telephone number available. If a telephone number is not available, a second reminder email is sent. |
2 weeks later |
Non-responders to either telephone follow-up or second reminder email will be removed from sample. A replacement person will be selected. |
1-4 weeks after email invitation to participate |
If potential participant replies with interest, a follow-up email is sent that someone will call to schedule interview. |
1-4 weeks after email invitation to participate |
If a potential participant does not reply, scheduler will call to schedule interview. |
2 business days after agreement to participate |
Email confirmation is sent. |
Data collection procedures for the expert panel are described below and presented in Table B.2-3:
The expert panel sessions(s) will coincide with the SCTC Annual Meeting to be held in April 2015. An email invitation will be sent one month prior to the April 2015 SCTC Meeting (Attachment 19). There will be up to two expert panels consisting of no more than nine participants in each group.
Table B.2-3 Procedures for Expert Panel
Timeline
|
Activity |
3 weeks after OMB approval |
NCI email study announcement sent to all potential participants |
March 2015 |
Email invitation letter sent to all potential participants for expert panel |
April 2015 SCTC Meeting |
Expert panel held during meeting |
Data will be gathered using three different modes of data collection: a self-administered online web survey (Attachments 1 and 2); telephone interview audio files (Attachments 3 through 7); and expert panel audio files (Attachment 8). The information collected will determine relationships, interactions, and connectedness among different network partnerships over time and with policy makers; assess the utility of research tools, interventions, products and findings; and determine key indicators for broad adoption of research products.
B.2.1. Quality Control
The contractor for this study will establish and maintain quality control procedures to ensure standardization, and high standards of data collection and data processing. The contractor will maintain a log of all decisions that affect sample enrollment and data collection. The contractor will monitor response rates and completeness of acquired data, and provide NCI with progress reports in agreed upon intervals.
B.3.1. Follow-up
To improve response rates and cooperation, follow-up efforts will be used to encourage survey and telephone interview completion. The efforts to reduce the number of non-respondents apply only to non-respondents and consist of the following:
Reminder emails will be sent to all non-respondents beginning one week after initiation of the survey and telephone interview. For the survey, a second reminder email will be sent one week after the first reminder email and, for the telephone interview, a second reminder email will be sent to those individuals for whom there is no phone number available for a telephone follow-up call. (Attachments 20-23).
One week after the second reminder email, telephone follow-up for non-response will begin for those individuals with a telephone number available. Experienced telephone interviewers, who are trained in non-response conversion, will make up to three attempts for the follow-up calls. The call is a prompting call only encouraging the potential participant to participate. The telephone interviewers will use the appropriate Telephone Follow-up Script for the call corresponding to the data collection mode - either survey or telephone interview (Attachments 24-25).
For the telephone interviews, if a participant breaks off the interview early, the interview will be considered as a “non-response” and the participant will be replaced with the next individual from the selected sample group. If a participant breaks off the interview near the end, the interview data will be retained.
B.3.2. Unit Nonresponse
As noted, to improve nonresponse, experienced interviewers who are particularly skilled in nonresponse conversion will contact non-respondents. If a potential participant is unable or unwilling to participate, refusal conversion will not be attempted. Tracing, using online searches, will be conducted for survey non-respondents for whom a telephone number is not available. For the telephone interview, the next potential participant in the strategic sample will be selected to participate and will be contacted following the usual contact procedures outlined in Section B.2 above.
B.3.3. Item Nonresponse
For the web survey, although our procedures are designed to maximize item response rates, the analysis will need to confront the issue of missing data. Experience with previous surveys indicates that some respondents will omit responses to some specific items (e.g., more burdensome items), although they may have provided most of the data required. By employing good survey data collection practices, we expect to minimize the amount of missing data on any single variable to a very low level. However, if item nonresponse is unexpectedly high for any of the key analytic variables, hot deck imputation and reconstruction imputation techniques will be used to estimate missing-item values.
Two methods were used to inform development of the web survey and the telephone interview and to refine the collection of information, to minimize burden and to improve utility. The data collection instruments have been reviewed by Westat researchers and NCI staff.
B.4.1. Review of Web-based Survey
After several rounds of internal review, the web-based surveys were reviewed by an Evaluation Committee comprised of a group of SCTC scientists and stakeholders and NCI representatives. A series of questions related to ‘trust’ were determined as not providing useful information and, in order to minimize burden and shorten the length of the surveys, were eliminated. The review performed by the Evaluation Committee helped inform the development of the surveys for the full scale evaluation.
B.4.2. Pretest of Telephone Interview
Westat conducted a pretest of the two main telephone interview instruments with seven individuals: three PIs and four affiliated partners. The pretest lasted from February to March 2014 and each participant was interviewed once. The affiliated partners were selected based on their strong collaboration with PIs and representation from three different organizations (advocacy group, government organization and university).
The interviews focused on the telephone interview guide questions and on capturing participants’ experience with the interview, including the time needed to complete the interview, questions or instructions that were confusing, and information that was difficult to provide. The interviews generally lasted about 40 minutes.
Summarized below are the key findings from the pretest and the modifications that were made to the telephone interview in response to the findings, where applicable.
Completion time: The average completion time was 40 minutes. Modifications made to the questionnaire following the pretest have not significantly altered its length, so the average time reported by pretest participants has been listed in the introduction as the estimated time needed to respond.
Overall feedback for Affiliated Partners Interview Guide: Feedback from pretest participants on the interview was that the interview was easy to understand, easy to answer, was not burdensome, and was straightforward.
Modifications to Affiliated Partners Interview Guide: One question was eliminated because it was found to be redundant; another question was moved to the conclusion section as it flowed best at the end of the interview; and the introduction was shortened.
Overall feedback for PI/Co-PI Interview Guide: Feedback from pretest participants on the interview was that the questions were clear and not confusing, there was no hesitancy answering questions, and the interviewees appreciated the opportunity to provide feedback. The interview was welcomed and not perceived as a burden.
Modifications to PI/Co-PI Interview Guide: Question wording for two items in the interview was fined-tuned to clarify terminology (i.e. affiliated partners and success).
The following individuals were critical in developing the research plan, the conceptual framework, survey questions, and sampling strategies underlying the SCTC Process Evaluation. Many of the same individuals will be involved with analysis once the data are collected.
Sophia Tsakraklides, Ph.D. Westat Principle Investigator for SCTC Westat 1600
Research Blvd. 301-738-3580 |
Michael Steketee, Ph.D. Senior Study Director Westat 1600 Research Blvd. Rockville, MD 20850 240-453-2603
|
Keith MacAllum, Ph.D. Senior Qualitative Researcher Westat 1600 Research Blvd. Rockville, MD 20850 301-610-8854
|
Martha Palan, M.S. Senior Study Director Westat 1600 Research Blvd. Rockville, MD 20850 301-738-3526
|
Elizabeth M. Ginexi, Ph.D. Project Officer Tobacco Control Research Branch Behavioral Research Program Division of Cancer Control and Population Sciences National Cancer Institute 9609 Medical Center Drive Bethesda, MD 20892-9761 240-276-6765
|
|
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | TABLE OF CONTENTS |
| Author | Vivian Horovitch-Kelley |
| File Modified | 0000-00-00 |
| File Created | 2021-01-26 |