Supporting_Statement_PartB_10-17-07

Supporting Statement for a New Collection RE: 2007 National Park Service Comprehensive Survey of the American Public

OMB Control Number: 1024-new

B. Collections of Information Employing Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When Item 17 on the OMB Form 83-I is checked "Yes", the following documentation should be included in the Supporting Statement to the extent that it applies to the methods proposed:

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

Telephone Interview

Respondents will consist of one adult (age 18 or older) selected from each participating household in the 50 states and the District of Columbia. The potential universe of contacts consists initially of all land-line telephone numbers in the US with an area code, three-digit prefix, and next two digits that includes at least one residential telephone (i.e., one-plus hundred-blocks of land-line phone numbers). This initial universe will also be augmented to include a sample of US cell-phone numbers. There are approximately 100 million households in the initial potential universe of contacts, with approximately 220 million adults.

Respondents are eligible for inclusion in the sample if they are adults (at least 18 years of age) who can converse in English or Spanish. In the 2000 survey, a quasi-random method of within-household selection of a respondent was used, namely a request to interview the adult in the household who had most recently had a birthday. We anticipate that random selection will be the preferred method in the future. For the present survey, 3 distinct methods of respondent selection will be used. In one subsample (a random one-third of participating households), respondent selection will be based on last birthday, for comparability with the 2000 sample. In a second subsample, basing respondent selection on next birthday will help to identify and offset some of the biases that might exist in the last birthday method. In the third subsample, strict randomization (using CATI software to randomly select, for example, the second oldest adult in the household) will allow an assessment of possible selection bias under either of the other two methods, and will ensure comparability with future iterations of the survey when this will likely be the only method of within-household selection.

The land-line sample will be disproportionately stratified so that 500 interviews will be completed in each one of the 7 administrative regions of the NPS. This will allow results to be analyzed and reported at the regional level. Another 500 interviews will be completed from the cell-phone sample. Individuals reached in the cell phone sample will be asked whether their household has a land-line phone and if so, they will not be interviewed (to ensure the statistical independence of this sample of respondents from the RDD sample).

The following table summarizes the sampling plan.

Table B1. Respondent Universe, Sample Sizes, and Expected Response Rates

Stratum	Respondent universe	Respondent universe size (estimated)*	Sample size	Expected response rate (RR3)	Estimated responses
National Capital Region	Adults speaking English or Spanish, in land-line households	Unknown (approx. 2 million; regional boundary indefinite)	500 Fall 500 Spring 1000 Total	50%	250 Fall 250 Spring 500 Total
Northeast Region	Same as above	53,819,000 (includes National Capital Region)	500 Fall 500 Spring 1000 Total	50%	250 Fall 250 Spring 500 Total
Southeast Region	Same as above	45,329,000	500 Fall 500 Spring 1000 Total	50%	250 Fall 250 Spring 500 Total
Midwest Region	Same as above	51,101,000	500 Fall 500 Spring 1000 Total	50%	250 Fall 250 Spring 500 Total
Intermountain Region	Same as above	30,034,000	500 Fall 500 Spring 1000 Total	50%	250 Fall 250 Spring 500 Total
Pacific West Region	Same as above	37,024,000	500 Fall 500 Spring 1000 Total	50%	250 Fall 250 Spring 500 Total
Alaska Region	Same as above	460,000	500 Fall 500 Spring 1000 Total	50%	250 Fall 250 Spring 500 Total
Cell Phone Users	Adults speaking English or Spanish, in cell-only households	Unknown (approx. 20 million, also included in regional figures above)	625 Fall 625 Spring 1250 Total	40%	250 Fall 250 Spring 500 Total

^{*US Census Bureau data on adult populations by state, as of 2003.}

Because response rates in telephone surveys have been declining nationally, the response rates in the table have been estimated conservatively at 50% for each of the 7 regions in the land-line sample, and at 40% for the cell-phone sample. Response rate can be defined in various ways; the figures in the table refer to the Response Rate #3 formula as defined by the American Association for Public Opinion Research. The University of Wyoming’s national RDD survey in 2006 for the America the Beautiful Pass project achieved an RR3 of 65% for the full survey, but with non-visitors to federal recreation lands ruled out as ineligible. The final report on the 2000 NPS comprehensive survey (which was not conducted by the University of Wyoming) reported an overall “completion rate” of 88%, but did not specify what formula was used to calculate this number; it seems likely that this figure is something other than RR3. In any event, the 40-50% response rates indicated in the table provide reasonably achievable targets.

While respectable by current US survey standards, response rates in this range present the possibility of non-response bias. Non-response bias can be a serious problem if both of two conditions are met: 1) the proportion of non-respondents in the target universe is high, and 2) non-respondents differ substantially from respondents on key demographic, behavioral, and/or attitudinal measures. For the present study, non-response bias will be assessed by comparing the sample results to known population demographics obtained from the US Census Bureau. In addition, the sample will be post-weighted in inverse proportion to the numbers of telephones and adults in the household (as reported by the respondents), and further weighted to bring the sample of respondents into line with the age, gender, race and Hispanicity proportions within each region, based on Census data. The University of Wyoming’s experience with the America the Beautiful survey suggests that non-response bias in demographic characteristics will be modest, yielding minimal differences between the weighted and unweighted results. Insofar as demographic characteristics are correlated with behaviors and attitudes, the post-weighting just described should adjust adequately for non-response bias. In addition to this plan, we will also attempt to have non-respondents answer two questions, which will allow for comparisons between respondents and non-respondents. Refusals are quite varied. There are the irate refusals, who receive no call-back. There are soft refusals, who receive a call-back, at which time an attempt is made to conduct the entire interview. If a soft refusal says “no,” a second time, these are termed “hard refusals,” and we make no other attempt to call back. However, for the soft refusal that says “no” for the second time, we will attempt to encourage them to answer a very limited number of questions that would take less than one minute. Questions #5 and #6 from the survey instrument will be used in order to get meaningful comparisons between respondents and non-respondents. Upon receiving a second “no” from a soft refusal, the interviewer will read the following script:

Interviewer: “I understand that you don’t want to take this survey, but just for our records could I ask you two questions that will take less than one minute of your time? No one will call back in either case, but your answers to these two questions will help us understand who is not answering our survey questions.”

If “yes,” proceed to questions

If “no,” terminate call-back

Interviewer: “The National Park System consists of all the units managed by the National Park Service, including national parks, historic and cultural sites, and national monuments. How many times in the past two years have you visited any units of the National Park System?

Interviewer: Please consider all your experiences to date with National Park System units, including national parks, historic or cultural sites, or monuments. Using a 10-point scale on which “1” equals “very dissatisfied” and "10" equals “very satisfied,” how satisfied are you with the National Park System?

Interviewer: “Thank you so much.”

By using this approach, answers from the hard refusals will give us the ability to make comparisons.

We will purchase the RDD and cell-phone samples from Marketing Systems Group (Genesys), a leader in the industry, who will apply their proprietary methods for achieving optimally efficient and representative sampling. The purchased sample will consist of about 20,000 telephone numbers, crudely pre-screened by Genesys to eliminate, insofar as possible, disconnects, businesses, and other known ineligibles in the RDD numbers. Ineligibles not identified through pre-screening will be further screened through the survey calling. The sample will be randomly subdivided into 10 waves before calling begins, and each wave will be thoroughly worked in the survey calling (10 callbacks or more per unreached number) before deciding whether the next wave is needed. This step will facilitate a higher response rate and more representative final sample by reducing the tendency to over-represent the households that are easiest to reach and by conclusively identifying as many of the phone numbers as possible as either eligible or ineligible for the survey.

Checks on the randomized telephone numbers will be conducted to ascertain whether any redundancy has occurred in the phone numbers. Redundant phone numbers will be removed, and the list of telephone numbers reconstituted to complete the wave.

2. Describe the procedures for the collection of information including:

* Statistical methodology for stratification and sample selection,

* Estimation procedure,

* Degree of accuracy needed for the purpose described in the justification,

* Unusual problems requiring specialized sampling procedures, and

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The procedure for information collection will be a telephone survey conducted from the fully equipped calling facilities of the Wyoming Survey & Analysis Center at the University of Wyoming. The questionnaire will be programmed with WinCATI software (Windows-based Computer Aided Telephone Interviewing) for administration by a trained interviewing staff.

The methods of stratification and sample selection have been described in detail in the immediately preceding section B(1). In brief, the initial sample will be a Random Digit Dialing sample purchased from Genesys, disproportionately stratified by NPS region and augmented with a cell phone sample. Within-household selection of a specific respondent will use 3 different approaches, including 2 quasi-random methods (to facilitate comparison with the 2000 results) and one strictly random selection (to anticipate comparison with future iterations of the NPS Comprehensive Survey).

To provide NPS with the information needed about the demographics, behaviors, and attitudes of visitors and non-visitors to units of the National Park System, the main focus of analysis will be estimating key percentages for the nation as a whole and for each of the 7 NPS regions. If the true (but unknown) proportion on particular characteristics of interest is around 50%, the statistical margin of error in estimating this proportion from the full national sample will be less than 2 percentage points, with 95% confidence (using the large-sample, normal approximation for the binomial distribution). At the regional level, the corresponding margin of error based on 500 completions per region will be about 4.5 percentage points. If the true proportion for some other characteristic of interest is around 10%, the margin of error at the national level will be about one percentage point, and at the regional level it will be less than 3 percentage points.

For some purposes, the estimates of interest will need to be calculated separately for visitors and non-visitors. Based on the 2000 results, it is anticipated that recent visitors will comprise about a third of the sample. Accordingly, for a true proportion of 50%, the margin of error for the visitor subgroup at the national level will be about 3 percentage points (with 95% confidence), while at the regional level it will be less than 8 percentage points.

The foregoing calculations ignore the effect on sampling efficiency (design effect) from the disproportional stratification and from weighting, both of which may increase margins of error somewhat compared to simple random sampling. It is not possible to assess the overall impact of these two factors until the data are in hand and the weights are known. However, in the University of Wyoming’s recent national RDD survey for the America the Beautiful Pass project, which used a generally similar sampling approach, the design effect was estimated at about 1.2. This means that the true margins of error would equal those obtained from a simple random sample that was about 20% smaller than the actual sample size. Since the margin of error varies with the square root of sample size, a design effect of this magnitude has very little impact on the error margin.

In short, the level of accuracy for the current survey is quite sufficient for the purposes of the research. Caution will need to be exercised when analysis gets down to the level of the visitor subgroup within region, but at all higher levels of aggregation the margins of error will be comfortably narrow.

In the main, the sampling issues here present no unusual problems; they are the same kinds of issues that are routinely confronted in surveying the American public by telephone. One slightly unusual feature of the present design concerns splitting the data collection into fall and spring calling periods. The 2000 survey was conducted entirely in the spring. However, since many questions in the survey refer to the respondent’s most recent visit to a National Park System unit, the approach used in 2000 might have produced a seasonal bias. That is, asking in the spring about the most recent visit might have yielded a disproportionate number of reports about winter visits. Conversely, asking only in the fall might give more summer visits. By including both fall and spring calling periods, we will be able to look for any systematic differences between the two seasons, and will largely offset any such bias by pooling the two subsets of data.

The period of this data collection effort is substantially less frequent than annual, thereby reducing respondent burden. It has been 7 years since the first NPS Comprehensive Survey, and it is anticipated that, after the current survey, a 5-year cycle will be adopted.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Several approaches will be used to maximize the response rate. First, the screener portion of the questionnaire is very short and straightforward, which should encourage respondents’ perseverance through that section. WYSAC’s trained and experienced interviewers will invite ambivalent households to complete at least the essential screening questions before deciding whether to continue.

The interviewers are also trained to code initial refusals as either “soft refusals” or “irate refusals.” The latter include explicit statements of unwillingness to participate, and will be considered as final call dispositions. Those telephone numbers will immediately be removed from the calling pool to ensure that they are not called again. Soft refusals, on the other hand, include unexplained terminations of the call, as well as vague expressions of lack of time at the moment, lack of interest in the topic, and so on. Numbers coded as soft refusals will be called back approximately one week later by specially trained refusal converters who will repeat the request for voluntary participation. In our experience, the completion rate on callbacks after a soft refusal is typically quite high. Often, a second call on a different day or at a different time of day will reach a different and more willing adult in the household, and/or will be received at a more convenient time. This means that the screener section, at least, can be completed to determine household eligibility and to initiate within-household selection of a respondent, with a good chance that the selected respondent will participate.

Besides these efforts at refusal conversion, the sample will be randomly subdivided into waves before calling begins. Each wave will be thoroughly worked in the survey calling (10 callbacks or more per unanswered number, varying the time of day and the day of the week) before deciding whether the next wave is needed. This step is essential in minimizing non-response bias by reducing the tendency to over-represent the households that are easiest to reach. Repeated callbacks also increase the response rate, by converting unanswered numbers into completed screeners, interviews, or other final dispositions. Further, the unused waves do not count against the response rate because—having never been called—they do not become part of the active sample.

One aspect of intensively working the sample will be to spread the calling out across days of the week and times of the day. This is another necessary step to enhance response rate and minimize non-response bias. Our WinCATI software will queue the calls by time zone, beginning on the east coast and ending with Hawaii. Based on our experience with other surveys, the heaviest call volume will be on Sundays, 5-9 p.m. and Monday through Thursday, 6-9 p.m. (all expressed as local times in the time zone of the call destination). However, we will also include 2 morning shifts of at least 2 hours each week, plus 3 afternoon shifts per week. One of these daytime shifts will be scheduled for every Saturday while the calling is on-going. Assuming that approval is received to start the calling by the beginning of October, we expect to work the sample continuously for approximately 8 weeks and conclude the requisite number of completions for the first wave by the end of November 2007.

As an additional precaution to enhance response rate and reduce non-response bias, the interviewing script will be translated into Spanish. The Spanish version will be available in WinCATI for use by selected bilingual interviewers, as needed, in households that require or prefer to be interviewed in Spanish.

While many respondents will correctly identify a park unit they visited in the last 2 years, many will give a local name or will not know that an area visited is actually a unit of the National Park System. If respondents reply that they have visited a national park unit, but cannot correctly identify that unit (judged by comparison with a master list of the 391 units of the National Park System available to each interviewer) attempts will be made to list several of the units in the respondents’ state or area to assist in identification. The prompt list will include a site that is widely visited, as well as a site that reports smaller number of annual visits. In this manner, the respondent will receive a prompt that suggests several possibilities, and hence, increases the likelihood of accurate identification. However, for analyses comparing the current results with those from 2000 (when this additional prompting was not attempted), the visitor subgroup will be limited to those who correctly identify an NPS unit without prompting.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The survey instrument was developed during fall and spring 2006/2007 using the 2000 NPS Comprehensive Survey as an initial guide. Development also included consultation with NPS stakeholders to suggest new questions and with an advisory committee formed specifically to comment on question wording, format, and other technical aspects of the survey. (The 3 members of the advisory committee are listed in section B5.) The draft survey was pre-tested on one focus group of 9 African-American participants who responded to an advertisement in the leading Denver newspaper. A substantially different small-group cognitive interview is planned with 9 or fewer participants in the western US to further refine questions #25, #26, #27, and #28. These are new questions about soundscape management, natural resource and recreation management that, to our knowledge, have not been asked of a national household sample.

We are seeking clearance to conduct additional developmental work in August and/or September 2007. This additional work would consist of: 1) another small-group cognitive testing, this time conduced in the eastern US; 2) a Hispanic American focus group, and 3) a telephone pre-test of the full survey instrument. Participants in the cognitive testing and focus group will be paid $70 to compensate them for their time and travel expenses.

Small-group Cognitive Interviewing

In addition to the first small-group cognitive testing on 9 participants, a second small-group cognitive interview will be conducted in the eastern US to further refine the 4 new questions described above. Participants in the cognitive testing will be recruited through advertisements in a local newspaper. The script for this test is included as ATTACHMENT F.

Hispanic American Focus Group

In addition to the African American focus group already held, a focus group of 10-15 Hispanic American participants (also in the eastern US) is proposed. The questionnaire will be given as a self-administered pencil-and-paper survey. Group discussion will be conducted after participants complete the survey. The discussion will address national parks in general and any problems associated with the use of parks by participants. Participants will be recruited through advertisements in a local newspaper. The focus group script is included as ATTACHMENT G.

Telephone Pre-testing

When the script has been programmed for the WinCATI software, and upon OMB approval, a telephone pre-test will be undertaken involving a maximum of 20 completed telephone interviews. A few numbers will be randomly drawn (leftover random numbers from an earlier DOI survey) and dialed until the desired number of interviews has been completed using the WinCATI program. This will help to identify any programming errors, typographical errors, display problems, or other matters in time to correct them before actual calling begins. It will also identify special training issues unique to this survey that will need to be addressed with interviewers before the full-scale calling gets under way.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The following members of the WYSAC project team have been involved in statistical aspects of the design:

Dr. Burke Grandjean, Professor of Statistics and Sociology, University of Wyoming, 307-760-5913

Dr. Patricia Taylor, Professor of Sociology, University of Wyoming, 307-399-8565

Dr. Bistra Anatchkova, Survey Research Center Manager, University of Wyoming, 307-760-3459

The following NPS experts have been consulted on substantive and statistical aspects of the design:

Dr. James Gramann, NPS Visiting Chief Social Scientist, 202-513-7189

Dr. John Dennis, NPS Deputy Chief Scientist, 202-513-7174

The technical advisory committee for the NPS Comprehensive Survey includes:

Dr. Suzanne Cook, Senior Vice President for Research, Travel Industry Association of America, 202-408-8422

Dr. Joseph O’Leary, Dean, Warner College of Natural Resources, Colorado State University, 970-491-6675

Darryll Johnson, NPS Coordinator, Pacific Northwest Cooperative Ecosystem Studies Unit, 206-685-7404

The contractor who will collect and analyze the information is the University of Wyoming, by and through its Wyoming Survey & Analysis Center (WYSAC). Contact information for members of the contractor’s project team is provided above.

LIST OF ATTACHMENTS:

Attachment A provides a copy of The Organic Act of 1916.

Attachment B contains a copy of the General Authorities Act of 1970

Attachment C contains the National Parks Omnibus Management Act of 1998.

Attachment D contains section 8.11.1 of the NPS Management Policies 2006—“Social Science Studies”

Attachment E contains the 60-day Federal Register notice

Attachment F contains the complete telephone interviewing script

Attachment G contains the cognitive interview script

Attachment H contains the focus group script

Attachment I contains the contact script for the survey

9