Supporting
Statement
Part B: Collection of Information Employing
Statistical Methods
FERTILITY
KNOWLEDGE SURVEY
Submitted to
Office
of Management and Budget
Office of Information and Regulatory
Affairs
Submitted by
Department
of Health and Human Services
Office of the Assistant Secretary
for Health
Office of Population Affairs
Contact Information:
Diane
Foley, MD
Deputy Assistant Secretary of Population
Affairs
Phone/Office: (240)
453-2888
Email: [email protected]
This page is intentionally blank.
Supporting
Statement
Part B: Collection of Information Employing
Statistical Methods
Fertility
Knowledge Survey
Contents
B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS 1
1. Respondent Universe and Sampling Methods 1
2. Procedures for the Collection of Information 3
3. Methods to Maximize Response and Deal with Non-Response 4
EXHIBITS
Exhibit 2–Power Analysis Assumptions and Inputs for Determination of Female and Male Sample Sizes 2
Exhibit 3–Consultations on Statistical Aspect of Survey Design 6
ATTACHMENTS
Authorizing Legislation A – 1
Fertility Knowledge Survey: Female & Male Versions B – 1
Federal Register Notices C – 1
Notice of RTI Institutional Review Board Study Approval D – 1
Email Invitation E – 1
Consent Form F – 1
Email Reminders G – 1
The Deal: Your Rights and Responsibilities as a KnowledgePanel® Member H – 1
Privacy Policy for KnowledgePanel® Members I – 1
Statement of Ipsos’s Commitment to Privacy and Data Protection J – 1
This page is intentionally blank.
Supporting
Statement:
Part B Collection of Information Employing
Statistical Methods
Fertility
Knowledge Survey
The U.S. Department of Health and Human Services (HHS) Office of Population Affairs (OPA) is requesting Office of Management and Budget (OMB) approval to conduct a web survey (Fertility Knowledge Survey). The Fertility Knowledge Survey is a new data collection. The Fertility Knowledge Survey will be administered online using a non-probability sample of panelists from two internet research panels (KnowledgePanel® and YouthPulse Panel) maintained by Ipsos.
Sampling Frame for Knowledge and Youth Pulse Panels. Ipsos uses an Address-Based Sampling (ABS) methodology to recruit new members into KnowledgePanel. Each quarter, a stratified random sample of addresses is selected to replenish KnowledgePanel. The sampling frame from which panel members are recruited is the universe of all U.S. residential addresses, secured from the latest Delivery Sequence File (DSF) of the U.S. Postal Service. Given the low overall response rate for the panel (7%) and the unknown and unquantified sources of bias (Ipsos does not supply sufficient information to determine the extent to which non-response might be correlated with any of the factors in which OPA is interested in understanding), the sample for proposed study will be treated as a non-probability sample. That means weights will not be used in analyzing or interpreting the results of the Fertility Knowledge Survey.
Sampling Method for Fertility Knowledge Survey. Ipsos will draw a stratified random sample of eligible members from the KnowledgePanel® and YouthPulse Panel and invite them to complete the survey. The overall sample will be stratified by sex and age with the goal of obtaining approximately 1,286 male respondents and 1,730 female respondents (i.e., completed surveys). Within each sex group, the target will be to obtain 50% in the age group 18–24 years and 50% in the age group 25–29 years.
Expected Cooperation Rate. The expected cooperation rate for this survey (35%) is the percentage of eligible panel members invited to take the Fertility Knowledge Survey that completes the survey. The number of the eligible population invited to complete the survey is calculated by dividing the desired number of completed interviews by the expected cooperation rate, which is based on Ipsos’s experience with similar studies. A small reserve buffer will be added to both the number invited and the number of completed responses collected.
Exhibit 1 presents estimates for the number of eligible population in the panels (i.e., potential respondent universe), the number of eligible panel members that will be invited to complete the survey, and the estimated number of invited panel members that is expected to complete the survey (i.e., sample size).
Exhibit 1–Ipsos KnowledgePanel® and YouthPulse Panel: Number of Eligible Population, Number of Eligible Panel Members Invited to Complete the Survey, and Estimated Number of Panel Members Who Complete the Survey
Respondents |
Number
of |
Number
of |
Estimated
Number of |
Females |
|
|
|
18–24 years |
4,597 |
2,471 |
865 |
25–29 years |
3,902 |
2,471 |
865 |
Subtotal |
8,499 |
4,943 |
1,730 |
Males |
|
|
|
18–24 years |
3,322 |
1,837 |
643 |
25–29 years |
1,889 |
1,837 |
643 |
Subtotal |
5,211 |
3,674 |
1,286 |
Total |
13,710 |
8,617 |
3,016 |
To obtain the desired sample size (i.e., number of completed surveys) of English-speaking females and males 18–29 years of age, and to have a sufficient sample size for detecting differences between two sex groups and two age subgroups (18–24 and 25–29), RTI conducted a power analysis to determine the desired sample size (i.e., number of completed surveys) of 1,730 female and 1,286 male respondents. RTI recognizes the non-probability nature of the sample – the power calculations are meant solely to identify suitable sample sizes for the project, assuming a probability-based sample.
Using the assumptions presented in Exhibit 2, we computed the sample size needed for detecting a minimal detectable difference (MDD) of 5 percentage points for the comparison of males and females, an MDD of 6 percentage points for comparing the two female age subgroups, and an MDD of 7 percentage points for comparing the two male age subgroups.
Exhibit 2–Power Analysis Assumptions and Inputs for Determination of Female and Male Sample Sizes
Assumption |
Female Sample |
Male Sample |
Sample size
|
N=1,730 18–24 years of age = 865 25–29 years of age = 865 |
N=1,286 18–24 years of age = 643 25–29 years of age = 643 |
Design effect |
1.25 |
1.25 |
Estimated % with correct answers for variables of interest |
25% |
25% |
Power (Type II) |
80% |
80% |
Alpha (Type I) |
0.05 |
0.05 |
Alternate hypothesis |
One-sided |
One-sided |
Minimum detectable difference |
6 points: 18-24 vs 25-29 5 points: female vs. male |
7 points: 18-24 vs 25-29 5 points: female vs. male |
To the extent possible, we used previous research to inform what assumption to use in the power calculation for the percentage (or proportion) of respondents with a correct answer for the variables of interest. While there are no directly comparable fertility knowledge studies of the U.S. population 18–29 years of age, a 2014 study1 that was based on a non-probability sample of females 18–40 years of age found that for many knowledge variables similar to the ones in our survey, the percentages of respondents who had correct knowledge clustered around 25% and 75%. This same study1 found that there were differences greater than 6 percentage points between women 18–24 and 25–34 years of age on knowledge about factors affecting fertility (e.g., woman’s age, man’s age, obesity, and timing of sex during menstrual cycle) and knowledge about reproductive biology (e.g., menstrual cycles, ovulation, and egg production). Based on this information, we assumed 25% (percentage with correct answer for variables of interest) for computing power (the power results would be the same for 75%). This is a reasonable mid-point for many of the proportions in the study and for the overall range (0%-50%).
Regarding men’s fertility knowledge, we found only one study that included American males (108 university students). This study2 had six knowledge questions about women and age-related fertility, and the percentages of male respondents who answered correctly were less than 14% for most items, and 38% for one. We used a larger (n=701) study3 based on a non-probability sample of Canadian males (18–50 years of age) that included questions about risk factors for male infertility. In this study, the percentages of Canadian men who correctly answered questions similar to those in our survey fell between 68% and 79%; there were no data on age group differences. Therefore, for the power calculation for the male sample we also assumed 25% (percentage with correct answer for variables of interest). Because of the limited number of male panelists in the target age groups who are expected to complete the survey, we estimated that we could count on achieving 643 males in each age group, which would give us an MDD of 7 points in comparisons by age group. Whether or not 6- or 7-point differences are conceptually significant will depend on the relationship.
Ipsos will begin online data collection after Ipsos has tested and the RTI team has approved the final online version of the OMB-approved survey.
Inviting eligible panelists to participate. Ipsos will invite by email (“assign”) the 8,617 eligible panel members (4,943 females and 3,674 males) selected to complete the survey (see Exhibit 1). When Ipsos “assigns” (i.e., eligible panelist selected to be invited) a survey to a panel member, the panelist receives a notification or invitation in their password-protected email account to notify them that a survey is available for them to complete. Panelists can also access their assigned surveys from their password-protected personalized landing page on the panel website. The text of the email invitation is presented in Attachment E. The invitation includes an FAQ section that will provide panelists with enough information on which to base their decision to participate. The FAQs will address questions about the survey sponsor and purpose, topics addressed, potential risks, voluntary nature of their participation, their right to skip questions, information about privacy and confidentiality of data, and contact information if they have additional questions.
Informed consent procedure. The invitation will include a custom link that, when clicked, will take the panelist to the consent page without requiring any further login or password verifications. Ipsos will obtain informed consent (Attachment F) electronically from all participants prior to their gaining access to the survey.
Data collection. The data collection period, including nonresponse follow-up, will be 14–21 days. During that period, survey respondents can break off and return to complete an interview during a second or later session.
Nonresponse follow-up. To maximize response, Ipsos will send up to two email reminders (Attachment G) to those who, after 3 days, have not started the survey or have started but not completed it.
Participant questions or issues. Panel members will have access to a support line (1-800-782-6899), maintained and staffed by Ipsos, to ask questions and communicate problems related to a study. The toll-free phone number for the panel support line is provided in the study email invitation (Attachment E), consent form (Attachment F), and the Privacy Policy for KnowledgePanel® Members (Attachment I). In addition, there is a Support Center email address ([email protected]) for panelists to communicate with Ipsos. Ipsos logs into a panel relations database for each contact made or received.
Survey participants who contact Ipsos with a question or concern about the study will receive contact information for the RTI principal investigator (PI), Dr. Christina Fowler, and the RTI IRB. In addition, if a study participant reports an adverse event or serious problem, Ipsos will promptly notify the RTI PI. In cases of an adverse event or serious problem, RTI will inform OPA. All permission, consent, and assent forms include contact information for the hotline, RTI PI, and RTI Office of Research Protections.
Quality control procedures. An Ipsos Quality Control manager will oversee the quality control process for data collection. Multiple quality control procedures performed at various stages include review, programming, and testing of the survey instrument; rollout of the survey on the internet platform; assigning/inviting panelists according to the sampling plan; and monitoring data collection and nonresponse follow up. Before, during, and after data collection, the RTI and Ipsos teams will meet weekly to monitor progress, troubleshoot, and ensure timely preparation and submission of the datafile, survey field report, and other documentation.
Several strategies will be employed to maximize response and deal with nonresponse. As noted in Section A.3, there are several advantages to a web survey that reduce burden, increase response, and increase data quality. We highlight some additional strategies that we will implement to maximize response rates, including
Use of the HHS logo in study materials to highlight the importance of taking part and of the information to be collected
Providing for a small survey-specific incentive (see Section A.9) that also serves to minimize breakoffs
Designing a user-friendly web survey that includes clear display of instructions, assurances (confidentiality and privacy), gentle warnings about the value of information to the study of key questions they have skipped, and “don’t know” or “not sure” response options
Allowing respondents to prepare respondents for potentially sensitive questions and to remind them of privacy and confidentiality assurances
Sending up to two email reminders (Attachment G) after 3 days when an invited respondent has not started the survey or has started, but not completed it
Extending data collection from the customary 14 days to 21 days
Item Nonresponse. To study the presence and size of item nonresponse, we will do the following:
Calculate univariate frequencies for all variables to determine the range of missingness rates.
Based on the importance of the variables, we will make a determination as to whether and how to deal with the missing data.
Cognitive and Usability Testing. As described in Supporting Statement Part A, Section A.8, in December 2018 RTI completed cognitive and usability testing of the online survey with nine males and females, 16-29 years of age, to assess whether they understood the survey questions, whether they could answer them as intended, and whether the visual layout, design, and navigational features of the online survey were easy to follow.
Testing consisted of 75-minute in-person, one-one-one interviews during which a trained interviewer used the concurrent think-aloud method of cognitive testing to solicit participants’ feedback through question-specific and retrospective probes. During the interview, the participant completed the survey online. The interviewer was able to view the screen of the laptop used by the participant via a separate monitor connected to the participant’s laptop. With permission of the participants, eight of the nine interviewers were audiotaped and observed (behind one-way mirror) by another member of the research team. Interviewers and participants were matched by sex, and participants received a $75 payment for their time.
Most participants understood the questions, key terms, and phrases and were able to provide appropriate responses. Some technical terms were unfamiliar to some participants, and other terms and phrases were not defined and were interpreted inconsistently. For most questions, participants found the response options adequate and suitable. Some participants noted that some questions were more sensitive, but they did not have difficulty answering them. Participants did not experience usability or navigation challenges.
In response to this feedback and in consultation with RTI survey methodologists, the RTI team changed the instrument by defining or describing key terms and phrases on every page where they appear; using the definitions of “female fertility” and “male fertility” instead of the terms themselves; defining vague terms and phrases; revising the instructions for true/false statements; and bolding or underlining words for emphasis.
Exhibit 3 supplies the name, affiliation, telephone number, and email address for each individual consulted on statistical aspects of the design and their role in design, collection, or analysis of the data. The list also includes the name of personnel responsible for receiving and approving contract deliverables.
Exhibit 3–Consultations on Statistical Aspect of Survey Design
Name |
Telephone
Number |
Role |
|||
Design |
Collect |
Analyze |
Other |
||
Jamie
Ridenhour, MS |
(919) 541-6567 |
|
|
|
Deliverable review |
Karol
Krotki, PhD |
(202) 728-2485
|
|
|
|
Deliverable review |
Christina
Fowler, PhD |
(919) 316-3447 |
|
|
|
Subcontract management, deliverable review |
Helen
P. Koo, DrPH |
(919) 493-1207 |
|
|
|
Deliverable review |
Michael
Lawrence, PhD |
(202) 370 6345 |
|
|
|
Subcontract management, deliverable review |
Carol
Ward, DrPH |
703-983-0388 |
|
|
|
Subcontract management, deliverable review |
Stefanie
Schmidt, PhD |
703-983-4074 |
|
|
|
Subcontract management, deliverable review |
Kate
Ahrens, PhD |
(240) 453-2802 |
|
|
|
Deliverable review |
Karen
Silver |
(240) 453-2802 |
|
|
|
Deliverable review and approval |
Nanci Coppola Office of Population Affairs |
(202) 690-7694
|
|
|
|
Deliverable review and approval |
Diane
Foley, MD |
(240) 453-2888 |
|
|
|
Deliverable review and approval |
Alicia
Richmond Scott |
(240) 453-2816 |
|
|
|
Deliverable review and approval |
* Dr. Ahrens left OPA in July 2018.
1 Lundsberg, LS, Pal, L, Gariepy, AM, Xu, X, Chu, MC, and Illuzzi, JL. (2014). Knowledge, attitudes, and practices regarding conception and fertility: A population-based survey among reproductive-age United States women. Fertility and Sterility, 101(3):767-774.
2 Peterson, BD, Pirritano, M, Tucker, L, and Lampic, C. (2012). Fertility awareness and parenting attitudes among American male and female undergraduate university students. Human Reproduction, 27(5):1375-1382.
3 Daumler, D, Chan, P, Lo, KC, Takefman, J, and Zelkowitz, P. (2016). Men’s knowledge of their own fertility: A population-based survey examining the awareness of factors that are associated with male infertility. Human Reproduction, 31(12):2781-2790.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |