Part B. Statistical Methods (used for collection of information employing statistical methods)
Respondent Universe and Sampling Methods
FDA plans to survey U.S. manufacturing establishments that produce cosmetics, also referred to as health and beauty products or personal care products. All U.S. cosmetics manufacturing establishments will be eligible for the survey and will be included in the sampling frame.
Stratified, systematic sampling will be used to ensure accurate representation of subpopulations of interest.
The sampling frame was built by FDA, combining data from four sources: FDA’s Voluntary Cosmetic Registration Program (VCRP), Dunn & Bradstreet (D&B), Information Resources, Inc., and Label Insight. The sampling frame includes information on each establishment’s size (based on the number of employees at the establishment), geographical region, establishment type (based on if they manufacture only cosmetics or tattoo ink or if they also manufacture drugs and/or medical products), address, telephone number, Standard Industrial Classification (SIC) codes, sales volume, description of their business activity, and if they are a public or privately held company.
Our target population is all U.S. cosmetics manufacturing establishments and our resulting sampling frame represents this target population.
To allow for a comparison of survey results among establishments of different sizes, the sample will be stratified by three size categories based on number of employees at each establishment (small: 1–19 employees; medium: 20–499 employees; and large: 500 or more employees). We stratify by establishment size because we believe the current manufacturing procedures likely differ for small, medium, and large establishments. In order to achieve the desired levels of precision for each strata, we apply the sampling formulas separately for each group. For large establishments, because there are so few of them, we need to take a census to achieve the precision we are aiming for. For small and medium establishments, we take a sample as described in the following section on precision.
A fourth level of the stratification variable will be all tattoo ink manufacturers. Because we were only able to identify a small number of these manufacturers, tattoo ink manufacturers are in a separate stratum; and a census of these manufacturers will be included in the sample to achieve the desired level of precision. Table B‑1 shows the number of establishments in the survey universe or population by the stratification variable size plus tattoo ink manufacturers. This information was obtained from the sampling frame.
Table B-1. Number of Cosmetics Manufacturing Establishments in the Survey Universe by Size
Size |
Number |
Small |
4,005 |
Medium |
1,142 |
Large |
118 |
Tattoo Ink |
21 |
Total |
5,286 |
Notes: small = 1-19 employees, medium = 20 to 499 employees, large = 500 or more employees, tattoo ink=all tattoo ink manufacturers regardless of size
An indication of the expected precision of sample survey estimates is the widths of 95% confidence intervals (also known as the margin of error) calculated for statistics of interest. Decisions about desirable sample precision involve a trade-off between the need for accurate data and the costs of obtaining it. Larger sample sizes yield greater precision, but larger sample sizes also increase the cost of data collection. Precision levels of +/− 0.05 percentage points and +/− 0.07 percentage points, for estimates by size (stratification variable), were considered. Precision of +/−0.05 percentage points, although desirable, would be prohibitively expensive. Thus, precision at +/–0.07 percentage points was used for the sample design. The sample design provides for sample sizes that are expected to yield precision of +/–0.07 percentage points or better for estimates by size of all proportions. The sample sizes were calculated assuming proportions of 0.5, which allow for precision requirements to be met for all proportions. Also, an estimated design effect of 1.2 was factored into the sample size calculations.
Based on the population totals from the sampling frame, our precision requirements, and design effects, we have determined 224 completed surveys are required for the small-sized establishments and 201 completed surveys are required for the medium-sized establishments. For the large establishments with only 118 in the total population, a census will be selected. Similarly, for tattoo ink manufacturers, a census of 21 establishments will be selected.
A systematic stratified random sample of cosmetic manufacturing establishments will be selected. As previously noted, the stratification variable will be establishment size based on the number of employees at the establishment. The required number of completes per stratification level will be adjusted upward for anticipated eligibility and response rates to determine the final sample size for each level of the stratification variable. The eligibility rate, assumed to be 80%, accounts for establishments with inaccurate information in the sampling frame, establishments that no longer manufacture cosmetics, or establishments that are out of business. The eligibility rate is based on our experiences of other similar establishment surveys. The sampling frame was created by FDA using data from their voluntary registration database in addition to several other data sources to capture any establishments that did not voluntarily register, thus justifying a relatively high eligibility rate.
We understand the importance of achieving a high response rate to help minimize nonresponse bias. Based on RTI’s experience conducting the pretest for this study and RTI’s experience conducting other establishment surveys using the same or similar data collection protocols, we anticipate a 70% response rate for the proposed data collection. During the pretest, there were challenges identifying the appropriate person to complete the survey and some contacted individuals refused to participate, so a response rate greater than 70% does not seem realistic based on this experience. Also, for a recent survey of the meat slaughter industry conducted by RTI, the average weighted response rate of 66% (ranged from 63 – 80% depending on plant size) for a 60 minute survey which used a similar data collection protocol. For these reasons, we are assuming a 70% response rate for the proposed data collection. Because the response rate will be less than 80%, following OMB protocols, we will conduct a nonresponse bias analysis (see Appendix 4 for details on the nonresponse bias analysis).
The sample design for the cosmetics survey is expected to yield 564 completed surveys (see Table B-2). For each stratum (establishment size plus tattoo ink manufacturers), information is provided on the population size, required number of completes, and the final required sample size. For the large and tattoo ink strata, the sample size required to achieve the desired level of precision will require sampling all establishments in the population (i.e., taking a census). Systematic sampling will be used to select the sample for the small and medium strata, as described below.
Systematic Sampling
Systematic sampling will be used to select the sample for the small and medium strata. The purpose of systematic sampling (instead of random sampling) is to ensure that samples selected adequately represent the entire target universe or population. Systematic sampling within each stratum forces each sample to include establishments with varying characteristics. With systematic sampling, establishments in the sampling frame are first sorted and ordered within each stratum by a set of appropriate characteristics. Once sorted and ordered, sample points are selected by choosing every nth establishment in the sorted and ordered list until the entire sample is drawn. The factor n is calculated as the universe size of the stratum, divided by the sample size for the stratum.
We will sort the sampling frame before sample selection by establishment type (those that manufacture cosmetics only; manufacture multiple lines of business including cosmetics, drug and medical; manufacture multiple lines of business including cosmetics, but no drug and medical manufacturing; manufacture tattoo ink), and geographic region (New England, Midwest, South, and West). This will facilitate a sample of cosmetic manufacturing establishments across the nation ranging in geographic location and establishment type. After sorting the frame, we will stratify by establishment size and select a systematic stratified random sample of cosmetic manufacturing establishments within the small and medium strata.
Statistical estimates will be generated by applying appropriate survey weights to the respondent record data. Appendix 4 describes the procedures for computing survey weights.
|
Cosmetic Manufacturers |
|
|||
|
Small |
Medium |
Large |
Tattoo Ink |
Total |
Population Size |
4,005 |
1,142 |
118 |
21 |
5,286 |
Required Sample Size a |
400 |
359 |
118 |
21 |
898 |
Required Number of Completes |
224 |
201 |
118b |
21 b |
564 |
|
|
|
|
|
|
a The required sample size accounts for 80% eligibility rate and 70% response rate in the small and medium strata. For the large and tattoo ink strata a census will be selected.
b For the large and tattoo ink strata, the number of establishments in the population are so small that a census will be selected. We can’t select extra samples to account for eligibility and response rates. We will do our best to obtain as many completes from these strata as possible.
Notes: Small: 1–19 employees; Medium: 20–499 employees; and Large: 500 or more employees; Tattoo Ink: includes all tattoo ink manufacturers regardless of size.
Procedures for the Collection of Information
The survey protocol is outlined below. Our design is based on best practices as outlined by Dillman’s Tailored Design Method (2007), which optimizes the mode and timing of contacts to minimize survey error and maximize response.
Step 1: Identifying a point of contact / survey invitation. Our data collection plan begins with an attempt to contact the person most appropriate to receive, and most likely to complete, the survey. During this call, we will establish eligibility of the facility (i.e., the facility is currently manufacturing cosmetic products), determine the preferred survey mode (web or hardcopy), collect the email address or mailing address for the manufacturer, and identify the correct point of contact (POC) to direct all future contacts.
Step 2: Survey Packet. Once a POC is identified, manufacturers will receive a survey packet via e-mail or mail, based on their preferred mode. Mail surveys will be sent via the U.S. Postal Service First Class mail within 48 hours. The packet will contain a cover letter, the survey instrument, and a postage-paid return envelope if by mail.
Step 3: Thank You Reminder E-mails/Postcards. After sending the survey invitation, manufacturers who have not yet completed the survey will receive thank-you/reminder e-mails and postcards. POCs who received the survey packet electronically will receive an email. POCs sent a hardcopy survey packet will receive a bifold postcard that also includes login credentials for the Web survey.
Step 4: Replacement Hardcopy Survey Packets. Approximately 4–6 weeks after the original survey packet, we will send a hardcopy replacement survey via U.S. Postal Service to all manufacturers that have not yet returned a completed survey. This mailing will include a cover letter encouraging participation, a replacement survey, and a return postage-paid envelope. The cover letter will include login credentials for the Web survey if they prefer to complete the survey online.
Step 5: Resend Thank You Reminder E-mails/Postcards. After resending the hardcopy survey packet, all manufacturers who have still not yet completed the survey will receive thank you/reminder e-mail or postcard to remind POCs to complete the survey. POCs who received the initial survey packet electronically will receive an email. The bifold postcard sent through the U.S. Postal Service will also include login credentials for the Web survey in case they change their mind and prefer to complete the survey online.
Step 6: Telephone Prompting Calls. RTI will contact by telephone all manufacturers that have not completed the survey after receiving the replacement survey to inquire about the status of their survey.
Appendices 5 and 6 include telephone scripts and survey materials that will be used during data collection.
Methods to Maximize Response Rates and Deal with Non-response
Achieving a high response rate is important to minimizing nonresponse bias. The data collection procedures employed by the contractor will be designed to maximize the response rate, including the following activities:
working with industry (for example, by meeting with trade associations) to secure their support of the survey;
securing establishment “buy-in” through clear and effective explanation of the importance of the study;
developing rapport and trust through effective and consistent messages conveyed from telephone interviewers to the individual respondents;
using a variety of methods and communication modalities to convey the importance of the study, including postcards, emails, and telephone calls;
developing a carefully designed and thoroughly tested survey instrument that is a reasonable length;
using highly trained individuals, outfitted with the most effective technological tools, to gain cooperation and minimize refusals in a timely and efficient manner;
operating a toll-free survey help line and an e-mail address that respondents can contact to request assistance when completing the survey;
sending a series of reminders and a replacement survey at carefully selected time periods after the first survey is sent;
releasing a reserve sample of establishments to recruit if eligibility is lower than anticipated;
ensuring the utmost confidence in the data security and privacy procedures in place by the survey contractor.
In addition, a nonresponse bias analysis will be conducted after data collection is complete.
Test of Procedures or Methods to be Undertaken
RTI conducted pretest interviews with six individuals from cosmetics manufacturing plants that varied in plant size and products manufactured. RTI recruited eligible plants and scheduled and conducted telephone interviews to pretest the survey instrument. The purpose of the interviews was to evaluate participants’ comprehension and interpretation of the survey questions and to identify unclear terminology, ambiguous phrasing, and inappropriate (or missing) multiple-choice response options.
Participants were sent a copy of the survey instrument to complete before participating in the telephone interview. During the telephone interview, the contractor recorded participants’ responses, probed for areas of difficulty, and asked a series of debriefing questions to assess participants’ overall understanding of the survey questions.
Overall the survey instrument was well received and understood. The pretest participants suggested several changes to the survey instrument, such as updating the response ranges for several questions, adding instructions for some questions or clarifying question instructions, and adding “not applicable” or “don’t know” as response options for some questions. We made the suggested changes and updated the formatting of the survey to make it more consistent. As part of the pretest, we asked participants to provide an estimate of the time required to complete it. The average time to complete the survey by all six respondents was 68 minutes. After the pretest, we made improvements to the survey format and survey questions, as noted above. Thus, we believe the average time to complete the current survey will be 60 minutes.
Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
The contractor, RTI International, will collect the information and analyze the data on behalf of FDA. Ms. Celia Eicheldinger (919-541-6222) of RTI International developed the sample design and estimation procedures. Carolyn Wolff, Ph.D., at FDA is the project officer for this contracted work.
References
Dillman, D. A. (2007). Mail and internet surveys: The tailored design method, 2nd ed. John Wiley & Sons.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |