Supporting Statement B for:
OPCRO,
DAIDS, NIAID, NIH, HHS
October 28, 2015
Name:
Lynda Lahl, RN, MS
Address: 5601 Fishers Lane, RM 9B25
Rockville, MD 20852
Telephone: 240-292-4887
Fax:
240-627-3111
Email: [email protected]
National Institute of Allergy and Infectious Diseases
National Institutes of Health
U.S. Department of Health and Human Services
Table of contents
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
B.1 Respondent Universe and Sampling Methods
B.2 Procedures for the Collection of Information
B.3 Methods to Maximize Response Rates and Deal with Non-response Bias
B.4 Test of Procedures or Methods to be Undertaken
B.5 Statisical Analysis Approach
B.6 Individuals Consulted on Statistical Aspects and Individuals Collecting
and/or Analyzing Data
List of Attachments:
Attachment 3: DAIDS staff T1 survey initial email invitation
Attachment 4: Extramural Researchers/External Stakeholders T1 survey initial
email invitation
Attachment 10: DAIDS staff T1 survey reminder email invitation
Attachment 11: Extramural Researchers/External Stakeholders T1survey reminder email invitation
Attachment 12: Email invitation for T2 web-survey participants
Attachment 13: Reminder email with unique link to complete T2 survey
Attachment 15: Recruitment Email to potential focus group participants
Attachment 18: DAIDS Staff Survey Screenshots
Attachment 19: Extramural Researchers/External Stakeholders Survey Screenshots
Attachment 20: Focus Group Opening Script and Questions
Attachment 21: CEPI Pilot survey results
We are interested in three target populations (i.e., DAIDS staff, extramural researchers, and external stakeholders), as well as a number of population subgroups (e.g., DAIDS Staff, contracted-DAIDS Staff, PPD-DAIDS Site Monitors, Network Extramural Researchers, Non-Network Extramural Researchers, International Extramural Researchers, Domestic Extramural Researchers, Pluripotent Extramural Researchers, International External Stakeholders, and Domestic External Stakeholders). This process assessment will include web- surveys at two time points (Time 1 and Time 2) and in-person focus groups. The target respondents for each group and sampling methods are addressed below.
DAIDS maintains an email distribution list for persons outside of DAIDS interested in receiving updates on DAIDS policies. This list has approximately 4,000 subscribers, and provides an efficient mechanism to reach the target populations of extramural researchers and external stakeholders. The distribution list includes email addresses only (i.e., there are no names, institutions, or other identifiers associated with the email addresses). DAIDS will send an email invitation (Attachment 3) and reminder emails (Attachment 11) to all subscribers on this list to participate in the web-survey at Time 1. The responders will be a convenience sample, as they will self-select whether or not they wish to participate. We will accept the first 400 responders. We anticipate a response rate of ten percent, based on the response rate to the pilot survey (Attachment 21). Of those that respond at Time 1, we expect an 80% survey click-through rate (i.e. 80% response completion rate). DAIDS will send an email invitation (Attachment 3) and reminder emails (Attachment 11) to all subscribers on this list to participate in the web-survey at Time 2. SSi will send an email invitation (Attachment 12) to all web-survey participants from Time 1 to invite them to participate in the Time 2 survey. Follow-up reminder emails (Attachment 13) will also be sent. We are interested in assessingchanges to awareness, accessibility, understandability, and applicability of the CEPI program, and changes in participant’s awareness, accessibility, understandability, and applicability of the CEPI program over time. over time, so we will not contact anyone that did not participate at Time 1. We anticipate a ten percent attrition rate for responders from Time 1 to Time 2. (See Table 1-1 for estimated number of ‘Extramural Researchers/External Stakeholder’ participants at each time point).
DAIDS maintains an email distribution list for its Full time Employees (FTE) and contractor staff. There are approximately 230 DAIDS staff members on this list. DAIDS will send an email invitation (Attachment 4) and reminder emails (Attachment 10) to all subscribers on this list. The responders will be a convenience sample, as they will self-select whether or not they wish to participate. We will accept the first 100 responders; we anticipate a response rate of forty percent, based on previous experience with this group. Of those that respond at Time 1, we expect an 80% survey click-through rate (i.e. 80% response completion rate). DAIDS will send an email invitation (Attachment 3) and reminder emails (Attachment 11) to all subscribers on this list to participate in the web-survey at Time 2. SSi will send an email invitation (Attachment 12) to all web-survey participants from Time 1 to invite them to participate in the Time 2 survey. Follow-up reminder emails (Attachment 13) will also be sent. We are interested in asessmingchanges to awareness, accessibility, understandability, and applicability of the CEPI program, and changes in participant’s awareness, accessibility, understandability, and applicability of the CEPI program over time. , so we will not contact anyone that did not participate at Time 1. We anticipate a ten percent attrition rate for responders from Time 1 to Time 2. (See Table 1-1 for estimated number of ‘DAIDS’ participants at each time point).
A limited number of DAIDS staff and extramural researchers/external stakeholders will participate in the focus groups. Focus groups will be conducted with in-person participants only and participation is limited to a single focus group. Please note that focus group participants can also participate in the web-survey. No more than 9 participants will participate in each focus group, and we anticipate conducting 9 focus groups consisting of a total of 81 participants. In our experience, focus group participation reaches a saturation point at nine participants (i.e., there is no added value to having more than nine individuals participate in a single focus group). DAIDS project staff will determine the location, date, and time of the focus group. Focus group interviews will take place in either Bethesda/Rockville, MD (for DAIDS FTE and contract staff) or at local (e.g., Washington, DC), domestic (e.g., Seattle, WA), or international (e.g., Cape Town, SA) network meetings (for extramural researchers and external stakeholders). Network meeting participants include persons from our target population (i.e., domestic and international extramural researchers, external stakeholders). DAIDS will obtain registration information from individual networks for persons registered to attend a DAIDS Network meeting. DAIDS will send an email to these individuals, inviting them to participate in a focus group being held in the same location as the network meeting at a time that will not interfere with their network meeting commitment (see Attachment 15). Persons interested in participating in a focus group are asked to contact the DAIDS project staff representative. DAIDS will email the informed consent form, which includes additional focus group details. All participants will be asked to complete and sign a consent form prior to participation, bringing the completed form with them to the focus group. Consent forms will be available for signature at the beginning of each focus group, in case a participant forgets to bring the signed consent to the focus group session. Social Solutions will verify completeness of the consent forms prior to the beginning of each of the focus groups to ensure there is no missing information and clarify questions participants may have. The same email distribution list used to invite DAIDS FTE and contract staff members to participate in the web-survey will be used to invite them to participate in one of two focus groups for DAIDS staff. The DAIDS staff focus groups will be held at or near their place of work.
Table1-1. Estimated number of participants at Time 1 and Time 2
|
Time 1 |
Time 2 |
DAIDS (n=100) |
80 |
72 |
Extramural Researchers/External Stakeholders (n=400) |
320 |
288 |
Web-based surveys using Survey Monkey will collect from the sample of 500 participants twice during the project period. Multiple persons from the same institution may participate in the survey (including investigators, site staff, and their supervisors) as the survey is collecting data on individual participants’ awareness, accessibility, understandability, and applicability of the DAIDS Critical Events policy. Time 1 (T1) will occur during the fall and/or winter of 2014 depending on the date OMB clearance is received. Time 2 (T2) will occur one year later during the fall/winter/spring of 20156. Each collection point will remain “open” online for three months, encouraging maximum participation. These two time points will allow measurement of any increases in knowledge of the policy and supplemental resources, increases in awareness and understanding of the policy, application of the policy, and participation in dissemination activities. See Attachments 18, 19, and 20 for DAIDS staff and Extramural Researchers/External Stakeholders survey questions and focus group opening script and questions.
Audio-recordings will be made for each focus group discussion. Focus groups will be moderated by Dr. Ami Lynch and transcribed by Mr. Jeremy Braithwaite, both of SSi. Meeting proceedings will be provided to NIAID/DAIDS staff with the necessary electronic and hard copies of the transcripts following each focus group, if requested. These transcripts will be uploaded to Dedoose for future analysis.
Non-response bias (i.e. survey respondent data is statistically different from potential respondents who would have taken the survey, but did not) is possible when survey response rates are low. In this study, response rates will be optimized at both T1 and T2 through the provision of incentives as well as email reminders at time points 1 and 2. The goal of the incentives is to encourage participation, effectively minimizing the potential for non-response bias.
We do not believe there will be a difference in the web-survey responders and non-responders, which would lead to non-response bias. The responders and non-responders will have commonality in their roles (extramural researchers, external stakeholders), and exposure to the CEPI program. We anticipate with an N of 400 there will be sufficient representation of each subgroup, as well as no differences from responders to non-responders.
In this process assessment, the response rate shall be calculated by determining the number of individuals who completed both T1 and T2 surveys (numerator figure) and scaling this by the total number of individuals recruited for the survey research (denominator figure). In order to maximize the response rate at both time points, NIAID staff will launch periodic follow-up reminder broadcasts with DAIDS staff, extramural researchers and external stakeholders. For survey point T1 and T2, NIAID will send reminder emails to all participants at three points (two to four-week intervals) following the initial blast recruitment email. For survey point T2, Social Solutions will send a reminder email to all T1 participants at three points (two to four-week intervals) following the initial blast recruitment email. A final reminder broadcast will be sent within one week prior to the survey closeout date. In addition to follow-up broadcasts, an incentive of $10 USD will be provided to survey respondents at both time points, increasing the likelihood that respondents who participate at T1 will also participate at T2. Though we expect an 80 percent response rate at both T1 and T2, we must also anticipate a 10 percent attrition between T1 and T2. By utilizing incentives and follow-up broadcasts, as well as offering modal survey heterogeneity (i.e. offering respondents the opportunity to complete the survey as a web-based, as well as hard copy format), we are confident that respondent attrition can be kept at the minimal 10 percent rate. This will yield a response rate of 72 percent, or 360 respondents.
A pilot-survey was conducted in late February/early March 2014 in order to field test and garner insights on the survey instrument. Of the 18 contacted individuals, four individuals (2 DAIDS FTE staff and 2 external stakeholders/extramural researchers) responded and completed the pre-test. The pilot-survey consisted of completion of the entire survey tool, in addition to Likert scale and free-response questions assessing survey burden, time to completion, clarity of instructions, clarity of survey questions, flow of survey, suggestions for improvement, and likelihood of participating in a longitudinal survey with offered incentives.
B.5 Statistical Analysis Approach
Social Solutions (SSIi) will use a basic level of analysis that will still allow studying the change in knowledge and attitudes surrounding the CEPI program at the group level, as opposed to the individual level. Unique identifiers will be collected from respondents at Time 2 in order to match them to Time 1 respondents. However, in the likely event that the same respondents at Time 1 do not complete the survey again at time 2, SSi will utilize nearest neighbor matching in order to artificially match respondents such that each Time 1 respondent is compared against a similarly matched respondent at Time 2. In order to match Time 1 respondents to Time 2 respondents, SSi will consider the following matching variables: role with DAIDS, DAIDS network affiliation, clinical research site type (e.g., domestic site, international site), years with clinical research experience, and whether the respondent is a native English speaker. Though SSi will consider all these variables in the matching process, priority variables will be role with DAIDS and years with clinical research experience, as these variables are likely very correlated with Critical Events knowledge, awareness, and understanding. After this data pre-processing is completed, bivariate statistics such as t-tests, chi-square tests, and McNemar’s test will be performed on the cleaned data set. Assuming there will be a similar response rate to the CEPI survey at Time 2, this approach allows the flexibility for accommodating small sample sizes, as well as response attrition from Time 1 to Time 2. Under the proposed method, SSi will employ an identical recruitment strategy at Time 2 as was employed at Time 1. Specifically, the entire population of DAIDS staff (N=179) and the entire population of extramural researchers/external stakeholders (N=4256) will be recruited by DAIDS staff at Time 2.
Regardless of the response rate at Time 2, SSi will conduct descriptive statistical analysis to describe the sample responses at both Time 1 and Time 2. This will consist of calculating measures of central tendency (i.e. mean, median and mode), in addition data visualization (i.e. generating bar charts, line graphs, etc.). In the event that the new strategy described above is not practical, Social Solutions will prepare a comprehensive descriptive analysis report consisting of measures of central tendency for variables of interest, data visuals demonstrating the magnitude and direction of change in variables of interest from Time 1 to Time 2, and cross-group comparisons (i.e. DAIDS staff vs. researchers/stakeholders). This method also allows the reader to gauge the change in variables of interest from Time 1 to Time 2; however, statistical tests of the significance of these changes will not be possible.
For the full T1 and T2 surveys, we anticipate estimating pooled cross-sectional time-series equations in which each individual (DAIDS staff member, researcher, stakeholder) is treated as a distinct unit. Because each individual contributes a single observation for each assessment point in the data set, it follows that each individual appears in the data set twice. These two observations per individual cannot be regarded as independent and the pooled cross-sectional time-series model takes this into account. More importantly, these models assess the degree to which changes in the independent variables are associated with changes in the dependent variables. Time-series models allow for much stronger causal claims as they help control for unobserved time-stable characteristics of the individuals and unobserved year-specific influences. This process eliminates potential bias resulting from the exclusion of any variable that either does not change over time or whose influence on the dependent variable does not change over time. Pooled cross-sectional time-series analyses can be modeled as either fixed or random effects. Fixed effects are most appropriate when omitted variables may vary across cases but are constant across time. Random effects offer more efficient estimators and can be useful when some omitted variables are suspected to be constant over time but vary across cases, while others are thought be fixed between cases but vary over time. The Wu-Hausman specification test can be used to determine whether the fixed effects (more conservative) or random effects (more efficient) models are more appropriate.
Jenny Namur Karp, MPH
Project Director and Principal Investigator
Social Solutions International, Inc.
Silver Spring, MD 20910
(202) 491-4954
Ami Lynch, PhD
Interviewer
Social Solutions International, Inc.
Silver Spring, MD 20910
(571) 212-8367
Jeremy Braithwaite, MA
Survey Analyst
Social Solutions International, Inc.
Silver Spring, MD 20910
(630) 621-6031
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement 'B' Preparation - 03/21/2011 |
| Subject | Supporting Statement 'B' Preparation - 03/21/2011 |
| Author | OD/USER |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |