0596-NEW FLMA Visitor Feedback SupStmt-B v11-10-14

The 2014 Supporting Statement - Part B for OMB 0596-NEW

Interagency Generic Clearance for Federal Land Management Agencies Collaborative Visitor Feedback Surveys on Recreation and Transportation Related Programs and Systems

B. STATISTICAL METHODS

Data collection methods and procedures will vary but be limited to the list of known Methodologies approved as part of this generic information collection clearance.

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The potential respondent universe will consist of FLMA visitors or users, potential visitors or users, recreationists, and members of stakeholder communities near FLMA units. Depending on the site, the survey may target special populations, such as subsistence users, who legally harvest plants, fish, game animals, and fuelwood. All study proposals submitted for approval under this generic clearance will include a description of that information collection’s particular respondent universe.

Information collections under this generic clearance may employ one of a number of sampling methods, based on the needs of the particular study. Random selection methods (including the next-birthday method) will be employed to obtain data that is representative of the study population. In addition, information collections may involve samples of self-selected customers, as well as convenience samples, and quota samples, with respondents selected either to cover a broad range of customers or to include specific characteristics related to certain products or services. The specific sample planned for each individual information collection and the method for soliciting participation will be described fully in each individual collection request made under this generic clearance.

We estimate that for every three year period there will be approximately 276,000 survey respondents and 61,800 respondents participating by other means such as focus groups, interviews or comment cards. Based on results of previous information collections at various FLMAs nationwide, it is anticipated that response rates will be at or above levels needed to obtain statistically viable results. Best Practices guidance will be provided to applicants on methods for maximizing response rates. In all cases, participation will be voluntary and time commitments will generally not exceed 20 minutes, though there may be a limited set of cases where participation will require 60-90 minutes (ie., focus groups).

2. Describe the procedures for the collection of information including:

* Statistical methodology for stratification and sample selection,

* Estimation procedure,

* Degree of accuracy needed for the purpose described in the justification,

* Unusual problems requiring specialized sampling procedures, and

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The FLMAs expect to use a variety of methodologies for these information collections as described in Part A. For example, survey staff may intercept visitors at FLMA sites and distribute hard copies of survey instruments for completion on-site. Alternatively, they may intercept visitors/users as they are leaving the FLMA units, if that is a more sensible approach for those units. The particular techniques will depend on the objectives of the study under investigation.

In addition to physical copies, information collection instruments may be electronically disseminated and/or posted on target pages of the respective FLMA’s web sites. In such cases, the FLMAs or their contractors may use commercial survey-specific software to automate collection and analysis of feedback. This method is likely to be combined with an in-person intercept, whereby visitors are intercepted at an FLMA site, their email information is obtained, and then they are sent a link to the online survey.

Telephone scripts, personal interviews, and focus groups with professional guidance and moderation may also be used, and are described in more detail below:

Telephone surveys: This method is most likely to be used for national population surveys or for populations that are geographically concentrated. Telephone numbers (including separate lists for cell-phone numbers) may be purchased for sampling purposes, and the survey may be conducted using Computer-Assisted Telephone Interviewing (CATI). It is possible that telephone numbers may also be acquired during an on-site intercept of visitors, though we do not anticipate that this combination of methods will be used much, if at all. Depending on the target population, it is also possible that telephone lists may exist (e.g. a club’s database of members), but again we expect few (if any) information collections will rely on pre-existing telephone lists.

Personal Interviews: This method will be utilized if qualitative, in-depth information needs to be collected. Visitors may be intercepted on-site and asked to participate in a brief interview (either right then and there or possibly at a later time, either in-person or by telephone). The target population for the interviews will vary depending on the objectives of the study, and the intercept method will be tailored accordingly. For example, if qualitative interviews of hikers are being conducted, the hikers might be intercepted and interviewed at the end of their hike. Finally, it is possible that interviews may be combined with the survey method, whereby interviewees are identified via the survey instrument. The survey would include a question asking respondents if they would be willing to participate in a brief interview to obtain more-in-depth information on their responses.

Focus Groups: Similarly, for focus groups, participants are most likely to be intercepted on-site (the location of the intercept will depend on who the target population is) and asked to participate in an on-site focus group either later that day or the next day. In addition, it is possible that focus groups may be conducted in cities across the U.S. (or in a specific city), if the objectives are to obtain in-depth, qualitative information from the general public or from a specific community (rather than from visitors of a particular FLMA site). Under such circumstances, focus group facilities will be used and their staff will recruit participants.

All submissions under this generic clearance will fully describe the relevant methodology, and will be carefully evaluated to ensure consistency with the intent, requirements, and boundaries of the anticipated programmatic clearance and to ensure that information-collection procedures are appropriate for the intended uses of the data. All information collection instruments will be designed and deployed based upon acceptable statistical practices and sampling methodologies, where appropriate, and will be used to obtain consistent, valid data that are representative of the target populations, account for non-response bias, and achieve response rates at or above levels needed to obtain statistically useful results. Proposed collection instruments and procedures must comply with OMB guidance as described in the OMB publication Guidance on Agency Survey and Statistical Information Collections (January 20, 2006).

Data collection methods and procedures will vary and the specifics will be provided with each collection request, including:

• respondent universe

• the sampling plan and sampling procedure (including stratification and selection methods for individual respondents)

• how the instrument will be administered to respondents

• expected response rate and confidence intervals

• strategies for dealing with potential non-response bias

• planned analysis

A description of any pre-testing and peer review of the methods and/or instrument will be provided with each information collection. In addition, all submissions under this generic clearance process will describe how data will be presented to managers and any others that will use the results of the information collections, particularly in cases where response rates were lower than anticipated. In these cases, program managers will take steps to ensure that the results will not be used inappropriately, for example, generalized outside the population of interest. Explanations will be provided with data presentations and reports so that users of the data understand any possible biases associated with the data.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Surveys must incorporate best practices to maximize initial response rates. The script used by the information collection staff to recruit participants will be designed to maximize response rates by emphasizing the importance of each respondent’s participation and by explaining how the information collection informs the improved management of their public lands. Visitors contacted in similar previous studies have often expressed gratitude and appreciation for the opportunity to participate in studies of this nature³. FLMAs will also be sensitive to survey length, in an effort to minimize respondent burden, and will use follow-up calls/emails as appropriate, to increase response rates.

Some visitors who are asked to participate in information collections will decline to do so. In such cases, measures will be taken to assess whether the accuracy of the collected data are affected by non-response bias, and if so, to correct for non-response bias effects on the data. All strategies for detecting and analyzing non-response bias will be included in the submission package. For example, the survey administrators may record observable information about every visitor contacted (e.g., group size, presence or absence of children in the group, time of contact). In addition, each visitor declining to participate in the study may be asked a few brief questions (e.g., their state or country of residence, is this their first visit to the particular location, and did they have trouble finding parking at the location). Thus, comparisons can be made between respondents and non-respondents to help establish the presence or absence of non-response bias.

If the results of this analysis suggest there is potential non-response bias in the data, additional statistical tests may be performed to assess whether there are non-response bias effects for key questions in the information collection (e.g., questions about visitor experience quality, mode choice, travel and parking issues). In cases where such bias exists, standard practices for weighting the data will be used (if feasible) to align important sample statistics with known population parameters. All procedures and results for the non-response bias analyses will be included in technical reports, and the likely effects of this bias (if any) on the interpretation of the data will be made clear to managers.

Measures will also be taken to address item non-response for those who agree to participate in the information collection. In particular, experienced survey administrators will attempt to convert “soft refusals” to completed information collections. Experienced analysts may also attempt to impute correct responses based on the respondent’s answer to other questions in the information collection. In cases where it is not possible to impute a correct response, the item will be treated as missing for that case.

As a result, measures to test and correct for non-response bias issues, coupled with the accuracy and statistical power associated with the projected sample size for each information collection instrument, are expected to result in levels of accuracy and reliability that are generally accepted as sufficient in peer-reviewed social science quantitative study findings. Interpretation of data will be made clear to all data users.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

Pre-testing of information collection instruments and procedures will be strongly encouraged as a means to reduce respondent burden and maximize the validity of the survey. If the number of pretest respondents exceeds nine members of the public, the affected FLMAs will submit the pretest instruments for review under separate cover, as part of this programmatic clearance. While most of the methods and questions submitted under this programmatic clearance have a long history of successful application and may not require extensive pre-testing, FLMAs will be encouraged to pre-test all survey instruments. This enables testing for respondent comprehension, the identification of sources of measurement error, and the estimation of burden hours. Ideally, participants in pre-tests should be drawn from the same respondent universe as the full sample. However, if this is not feasible, a similar respondent universe should be used.

5. Provide the names and telephone numbers of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The names and contact information of the responsible FLMAs and the principal investigator(s) who will collect and analyze the data will be included on all submission forms received under this generic clearance. Each information collection project will be peer reviewed by social scientists and statisticians in order to insure that the survey instrument and the methodology are designed in such a way as to provide scientifically valid data. The FLMAs will include the names and contact information of persons consulted in the specific information collection requests submitted under this generic clearance.

Of the FLMAs involved in any particular Information Collection, the lead agency will assign their Paperwork Reduction Act (PRA) officer to conduct an administrative review of the request and send the submission through its normal peer review channels on behalf of the participating agencies of that Information Collection. The PRA Officers for each of the agencies includes the following:

Agency	PRA Officer
Army Corps of Engineers	Christy King
Bureau of Land Management	Jean Sonneman
Fish and Wildlife Service	Hope Grey
Forest Service	Wolf Cota
National Park Service	Madonna Baucum

This review will insure that all documentation is complete and sufficiently detailed to meet the requirements of the PRA and scientific validity. The package will then be sent to the Forest Service for submission to OMB.

This programmatic clearance, including the compendium of survey questions and methodologies, is the product of a collaborative effort among the FLMAs. Indeed, each of the partner agencies has at least one, and in most cases a couple of representatives who have been actively participating in the project. The core team includes:

Agency	CVTS Representative(s)
Army Corps of Engineers	Kathy Perales, Meredith Bridgers
Bureau of Land Management	Rob Perrin, Randy Goodwin
Fish and Wildlife Service	Kevin Kilcullen
Forest Service	Wolf Cota, Marie Messing
National Park Service	Bryce Lloyd, Jennifer Kovarik, Paul Schrooten, Kerri Cahill

In addition, the following outside agencies have participated as part of the core team:

• Volpe National Transportation Systems Center (U.S. Department of Transportation)

• Bureau of Transportation Statistics (U.S. Department of Transportation)

• Federal Highway Administration (U.S. Department of Transportation)

• Paul S. Sarbanes Transit in Parks Technical Assistance Center

• US Geological Survey

This team has developed a charter to guide its work, convenes monthly calls to review progress and key deliverables, and has convened two planning workshops. This team has provided input on all the documents included herein.

1Vaske, J. J. Information collection research and analysis: Applications in parks, recreation, and human dimensions. State College, Pennsylvania: Venture Publishing Inc. 2008.

² Lawson, Steve; Ph.D.; Director, Public Lands, Resource Systems Group, Inc. Personal communication,September, 2012.
³ Ibid.

Page 1 of 4