Supporting Statement B

Information Collection Request

Reinstatement

Evaluation of Effectiveness of NIOSH Publications:

NIOSH Customer Satisfaction Survey – 5-year follow-up

Supporting Statement Part B

Submitted by:

Vernon P. Anderson, Ph.D.

Education and Information Division

National Institute for Occupational Safety and Health

4676 Columbia Parkway

Cincinnati, Ohio 45226

Phone (513) 533-8319

Fax (513) 533-8465

e-mail [email protected]

January 2008

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

Description of Universe: As already indicated, the target audience for this survey consists of occupational safety and health professionals who belong to one or more of the four primary professional organizations: AIHA (10,600 members), ACOEM (4,952 members), AAOHN (10,000 members) and ASSE (32,000 members/7,543 of whom are Certified Safety Professionals). Each organization will make their directories available to Battelle for sample selection. Of the largest group (ASSE), only a segment labeled as “certified safety professionals” (CSPs) (N=~7000) are likely to be engaged in workplace safety and health issues (other members are involved in fields such as product safety, traffic safety, aeronautical and marine safety, etc.) and will be targeted for inclusion in the survey. The ASSE will identify such individuals when developing the mailing list for the survey. For the sample selection, each professional group will be treated as a stratum. As an individual can belong to more than one professional group or stratum, a de-duplication operation at the frame level will take place. Individuals appearing on more than one membership list will be assigned to the group most aligned with their professional training based on the title listed on their membership address. NIOSH employees who are members of the four associations will also be excluded from the membership listings prior to sample selection.

Sampling Methods: Since the opinions of each profession about the topics in the survey – namely, sources for accessing NIOSH information, the use of NIOSH information in educational/training settings and program and policy development, the frequency of use of NIOSH publications, ratings of the NIOSH information delivery system and the content of NIOSH publications, and NIOSH outreach initiatives – is of equal importance to NIOSH, the designated total sample size of 1200 will be composed of an equal number of persons (300) drawn at random from each of the four associations. Equal sample sizes for each group were chosen because comparisons and estimates for subgroups were judged to be very important and equal allocation will optimize this objective, i.e. provide the greatest power. Using an equal allocation to strata will have an effect on the precision of our estimates of proportions for the four professional groups combined because the strata sampling fractions are unequal.

Sample Size Justification: Since the analysis of the results of the questionnaire will consist of descriptive statistics, the sample size was designed to produce summary measures estimated with a specified level of precision. Most of the summary statistics will be proportions, e.g. proportion of respondents who have taken a course or attended an educational program in which NIOSH publications were used. From the results of the 2003 Customer Satisfaction Survey – NIOSH Publications and Information Services, the majority of the percentages associated with the key findings ranged from 70 to 90 percent. Accordingly, the proposed total sample size of 1200 individuals, yielding an expected 720 completed questionnaires, is designed to produce 95% confidence intervals that are + 3.5 % in width for a proportion of interest close to 0.70.

Table B.1-1. Respondent Universe and Sample by Association

	AIHA	ASSE	ACOEM	AAOHN	Total
Population size	10,600	7,543*	4,952	10,000	33,095
Number of surveys mailed	300	300	300	300	1200
Expected number of surveys returned	201	189	156	174	720
Expected response rate	67	63	52	58	60
Actual response rate in 2003 survey	64	60	49	55	57

*The sample from ASSE includes only the 7,543 Certified Safety Professionals.

In our computations we have accounted for the fact that the differential sampling fractions in the strata will result in an increase in the sampling variance relative to proportional allocation to strata. The actual width of confidence intervals for a given sample size depends upon the magnitude of the proportion and as the proportion becomes closer to 0.50, the margin of error will increase. In the computation of sample size, we have assumed that the proportion follows a normal distribution. Accordingly, the 95% confidence interval for a proportion p will be of the form: of sample size, we have assumed interval for a proportion p will be of the form:

________

p + 1.96 x (p(1-p))/n) x VIF where

p = the estimated proportion

VIF = variance inflation factor because of the use of disproportionate stratified

sampling

Given the population sizes of the four organizations and approximately equal strata variances, we estimate that the equal allocation will result in a value of 1.07 for the variance inflation factor.

Because the selection probability for an individual will depend on his or her stratum, survey weights must be developed for the estimation of the proportions for the overall population. The base weight for a respondent will be the reciprocal of his or her probability of selection. In addition the survey weight will include a nonresponse adjustment factor that will be computed with the objective of reducing the bias in the estimates due to nonresponse.

In the event that statistical comparisons are desired among subgroups of respondents, contingency table analyses will be employed. Chi-square tests will be used to determine whether the distribution of responses between one or among subgroups are significantly different. When the subgroups are defined in terms of many characteristics, logistic regression models could be used to explore differences between professional subgroups.

2. Information Collection Procedures

Statistical method for stratification and sample selection: The project involves the mail-out and return of a self-addressed, postage-paid questionnaire form, with the option of completing the questionnaire electronically on a study-specific website. The study sample will include 300 randomly selected members of the four professional associations, for a total sample size of 1200. As noted in Section B.1., based on the results of the 2003 NIOSH Customer Satisfaction Survey, the majority of the percentages associated with the key findings ranged from 70 to 90 percent. Thus, the proposed sample size of 1200 individuals, yielding an expected 720 completed questionnaires, is designed to produce 95% confidence intervals that are + 3.5 % in width for a proportion of interest close to 0.70. In our computations we have accounted for the fact that the differential sampling fractions in the strata will result in an increase in the sampling variance relative to proportional allocation to strata, as detailed in Section B.1.

Sample selection is straightforward for this project. Since the satisfaction of each of the four professions is of equal importance to NIOSH, the designated total sample size of 1200 will be composed of an equal number of persons (300) drawn at random from each of the four associations. Equal sample sizes for each group were chosen because comparisons and estimates for subgroups were judged to be very important and equal allocation will optimize this objective, i.e. provide the greatest power. To select the random sample for each of the four associations, we will generate a random number for each record on a membership roster. The membership roster will be sorted according to the assigned random numbers, and the first 300 records on the list will be the sample for that roster. If a letter is returned undeliverable, the next record on the list will be used to replace the undeliverable record.

Data collection procedures: Battelle will conduct all data collection activities for this project. Using NIOSH letterhead, Battelle will print, prepare, and mail advance introductory letters to all participants (Attachment C2). These personalized letters will introduce the survey, explain how the participant was selected, alert him/her that paper surveys will be mailed shortly, and provide an opportunity for participants to choose to complete their survey via the web instead. A unique participant password (access code) will included in the letter, along with the study web address (URL) and the pre-addressed, post-paid survey response card (Attachment C3). The password will not be linked to the individual participant, to assure anonymity of response. A toll-free phone number will be included so that the participant may contact study staff in the event that they have any questions or concerns. As in the initial 2003 survey, the 2008 advance mailing will include a letter of support from the president of the participant’s professional association (Attachment C4). In addition, the packet will also include the NIOSH CDs as an incentive.

Sample members who return a survey response card will be removed from the list to be contacted in the follow-up mailings. The unique ID on this card will identify the sample member, but will not be linked to a specific survey response. Two weeks after the advance mailing, a full survey mailing will be sent to sample members who have not returned the Survey Response Card. The mailing will include a survey cover letter (Attachment C5), the letter of support from the president of the participant’s professional association (Attachment C4), and another pre-addressed, post-paid survey response card (Attachment C3). A second unique participant password will be included in the letter, along with the study web address (URL). A toll-free phone number will be included so that the participant may contact study staff in the event that they have any questions or concerns. Completed paper questionnaires will be returned to Battelle via business reply U.S. mail.

Two weeks after the full packet is mailed, Battelle will mail reminder/thank-you postcards to the entire sample (Attachment C6). The cards will contain a message to thank those who have already participated and encourage non-respondents to participate. The toll-free number established for the study will be included on the postcard, in case the respondent has questions.

Four weeks after the initial package is mailed, a second full package will be mailed to all non-responders, with a final version of the cover letter, the follow-up mailing cover letter (Attachment C7). Throughout the data collection period, Battelle staff will be available by phone via the toll-free study phone number to any participants who need assistance or have concerns that need to be addressed.

Quality control procedures: Quality control procedures will be incorporated into the data processing system through the use of TeleForm^®, a system which provides advanced optical character recognition (OCR) and throughput capabilities with accuracy rates approaching 100%. Battelle will develop robust forms that include range checks, skip pattern checks, and cross checks that are applied when scanning and verifying forms. In addition, Visual Basic scripts will be included in the web e-pdfs to incorporate quality assurance/quality control (QA/QC) features.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Based on the results of the baseline survey and considering protocol improvements designed to improve response rates, it is expected that 720 (60%) will complete a survey within the 12-week data collection period. Based on the results of the baseline survey, we anticipate that approximately 12% of the total responders will fill out their survey online with the remaining 88% completing a paper version. Table B.3-1 indicates the expected response rate by association.

Table B.3-1. Expected response rate by Association

	AIHA	ASSE	ACOEM	AAOHN	Total
Number of surveys mailed	300	300	300	300	1200
Expected number of surveys returned	180	180	180	180	720
Expected response rate	60	60	60	60	60
Actual response rate in 2003 survey	64	60	49	55

Plans to maximize response rates: A variety of measures have been taken to assure a high rate of returns to this survey:

1. Letters from the four partner associations will accompany the survey forms sent to each of their members in offering support for the project and encouraging participation. These letters will reinforce the importance of the survey to the members of each association.

2. The four professional associations have agreed to announce and promote the survey in advance of the mailing through their respective newsletters and house publications. This will help increase awareness of the survey and should increase the number of sample members that recognize the mailing and take the time to respond.

3. CDs of popular NIOSH scientific information products, including a nanoparticle library, and a CD version of the NIOSH Pocket Guide to Chemical Hazards will be included in the initial mailing as an incentive to all of those who participate. As noted earlier, research indicates that an incentive enclosed with the survey has a stronger impact on response rates than incentives provided upon completion of the survey. Pilot test participants responded very favorably to these CDs and felt that they were appropriate and desirable incentives for the target participants.

4. An attractive, well-designed, and user-friendly questionnaire addressing topics of importance to the target sample will facilitate response. The questionnaire has been formatted for ease of response by all participants. It has also been shortened considerably (approximately 22%) from the baseline version to reduce respondent burden.

5. In addition, several questions and response categories have been added to enhance the salience of the survey to the ACOEM membership in an effort to increase the response rate in this group, which had the lowest response rate in the initial survey.

6. Individuals will be give the option of responding to the questionnaire electronically via the web if they so choose, making it easier for those with access to the internet to complete the survey. Multi-mode surveys with an Internet option have been shown to yield higher response rates than surveys using only a single response option¹.

7. Follow-up mailings will be conducted to enhance response rates. It is well-established that higher mail survey response rates are achieved when repeated contacts, in the form of follow-up letters and appeals are employed². Therefore, in addition to the pre-survey advance letter and the initial survey packet, we will mail reminder/thank you postcards to the entire sample and a second copy of the questionnaire to all sample members who have not returned a Survey Response Card within six weeks of the initial survey mailing.

Plan to evaluation potential nonresponse bias: Two approaches to the computation of nonresponse adjustment factors will be considered and both approaches assume that there are characteristics for both respondents and nonrespondents that are associated with the propensity to respond and the key survey statistics. If there are only a few potential characteristics on the sampling frame for nonresponse adjustment, the weighting class method may be used. A cross classification of the characteristics would be constructed and in each cell of the cross classification, a nonresponse adjustment factor would be computed. The nonresponse adjustment factor would be the sum of the base weights of all sampled cases (respondents plus nonrespondents) in the cell divided by the sum of the base weights of all respondents in the cell. If there are several potential characteristics on the sampling frame that could be used for nonresponse adjustment, consideration could be given to building a response propensity model using logistic regression. Here the dependent variable is whether or not the sampled individual completed a questionnaire. After estimating the parameters of the logistic regression model, the predicted likelihood of response could be computed for each individual. The inverse of the predicted likelihood of response would then be applied to the base weight.

4. Tests of Procedures or Methods to be Undertaken

The data collection procedures and the majority of the questions on the survey were drawn from the baseline survey (OMB No. 0920-0544) in 2003. The majority of the questions were drawn from the baseline survey, as detailed in Attachment E.

A pilot test was conducted with a convenience sample of eight participants, two representing each of the four target occupations. Participants were sent the draft survey and asked to complete the survey and note any questions or response categories that were confusing. They were specifically asked for comments on the questions that had been added to the follow-up survey (i.e., those that did not appear on the baseline survey).

Pilot participant comments were gathered and reviewed by the NIOSH Technical Monitor and Battelle staff, and a number of changes to the questionnaire were incorporated into the final version. In addition, the timing of the survey was assessed during the pilot test. Most participants took about 20 minutes to complete the pilot survey. Following the pilot, several questions were eliminated to ensure that the average survey length would not exceed 20 minutes. The final survey was reviewed and approved by NIOSH personnel prior to submission to OMB.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Dr. Vern Putz Anderson, Research Psychologist from the Education and Information Division NIOSH, (4676 Columbia Parkway, Cincinnati, Oh 45226- Phone (513) 533-8319), Technical Monitor, oversaw all aspects of project design. Dr. Anderson is responsible for receiving and approving contract deliverables. Mr. Charles Wolters, a statistician from Battelle Centers for Public Health Research and Evaluation (6115 Falls Rd Ste 200, Baltimore, MD 21209; Tel (410) 372-2732) designed the sampling plan and will analyze the results of the survey. Ms. Marianne Story-Yencken (Battelle Seattle office, 1100 Dexter Ave N, Suite 400, Seattle, WA 98109; Tel (206) 528-3164), Certified Industrial Hygienist, is the primary liaison with the four partner associations. Dr. Lisa John, Project Director at Battelle Centers for Public Health Research and Evaluation (10420 Old Olive Street Road, Ste 300, St. Louis, MO 63141; Tel (314) 993-5234) will oversee the random selection of the sample of respondents, the printing and mailing of the forms, and the collection and processing of the returns. Dr. John and Mr. Wolters, both of Battelle, will analyze the data and prepare reports summarizing the results of the survey.

1 McMahon S, Iwamoto M, Massoudi M, Yusuf H, Stevenson J, Da Chu S et al. Comparison of e-mail, fax, and postal surveys of pediatricians. Pediatrics 2003;111:299-303.

2 Dillman, D. (2000) Mail and Internet Surveys. New York: John Wiley & Sons, Inc.