0596-0240-STEW-MAP-Supporting Statement B

Supporting Statement B

The Stewardship Mapping and Assessment Project (STEW-MAP)

OMB Control Number 0596-0240

Note: This request is for the renewal of the previously approved information collection OMB 0596-0240, the Stewardship Mapping and Assessment Project (STEW-MAP), expiring August 31, 2018. The USDA Forest Service requests approval from OMB to continue the collection of information from members of stewardship groups in multiple study areas across the United States.

B. Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corre­sponding sample are to be provided in tabular form for the uni­verse as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

There are three phases in this collection:

Phase 1 is a census to determine the universe of civic stewardship groups;

Phase 2 is the primary STEW-MAP survey implementation; and

Phase 3 (which is optional) is a set of follow up interviews with groups identified as major nodes in the social network analysis.

Phase One: Census to Determine Respondent Universe

The respondent universe for this collection will determined by taking a census of all known stewardship organizations in a designated study area. Using this method will provide a true measure of the population and will help to minimize sampling error. We will use the following methods:

1. Consolidate a list from the STEW-MAP partners in a location of key local technical assistance providers/known stewardship organizations in each of the study areas. In most cases, the research team will have prior relationships with the large known stewardship organizations in the target area and will have previously discussed the STEW-MAP project with them. This list is likely to number about 20 organizations.

During a brief telephone call designed only to request contacts, we will ask each group contacted to provide the contact information of at least 50 stewardship groups.

2. Conduct online searches to find additional organizations that are not on the list provided by those organizations contacted in step 1.
   
   

3. After we consolidate the lists we will create the “master list”. We will start by removing any duplicate entries and defunct groups and bad addresses. Then we will determine if we reached our goal of 1000 groups in the study area. If we do not reach our goal of 1000 groups we will use a modified Delphi method to determine if any additional groups are in the area but were not recognized by the first 20 contacts or found in our web search. We will use the “master list” to randomly contact 20 stewardship groups. We will ask them to provide names and contact information of at least 25 known stewardship groups in their area. We will conclude this process when the groups on the list are repeated and no new groups are added. We recognize and acknowledge that the research effort will vary in each sample area, and in the cases where the Delphi method is used the total respondent burden would increase by 5 hours per region. (See the table below.)
   
   

4. If we do not reach 1000 names, but are no longer learning of new stewardship groups through the process described above, we will conclude that there is a smaller universe of stewards in this particular location.

Activity per Region	Number of Contacts	Burden (minutes)	Total Burden/per region (hours)
Initial Phone Call	20	15	5
Second Phone call	20	15	5

Phase 2 – Survey

All of the organizations on the master list will be contacted and asked to participate in the STEW-MAP survey. This list will be the complete universe for this study and as such, the results will not be generalizable outside this universe of respondents.

Phase 3 – Follow-up interviews

These interviews are optional given local STEW-MAP team resources and needs, as described in Part A. We will select at least 10 organizations per region for follow-up interviews based on answers provided to the social network analysis questions on the survey.

2. Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

There will be no attempt to stratify this sample once the entire universe is established. This is a census survey and every organization on the master list will be contacted and asked to participate in the survey. There are no plans to generalize this data to a population outside this universe of organizations.

In Phase 3 (follow up interviews with primary network node organizations) we will select up to 10 organizations based on answers provided by all respondents to the social network questions on the survey. Occasionally, a location undertaking STEW-MAP may have specific needs that would warrant including a group seen as prominent in the civic stewardship of the area that did not show up as a major link in the network.

STEW-MAP data will initially be collected one time in designated cities and depending on the need for additional data, the same or similar information may be collected at 5- or 10-year intervals. Multiple data collections from the same population will allow longitudinal analysis of stewardship data for a given city to determine any changes in the structure of civic stewardship work and groups over time. This will also allow previous respondents to update their information on the online maps and allow new or additional groups to participate in the project.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.
   
   
   

We will follow Dillman, Smyth, and Christian’s (2009) guidance for maximizing the response rate on web-based surveys (invitation letters sent via email, but otherwise following the mail survey protocol, follow up reminders, and finally personal outreach via phone to those still not completing the survey).

We will mail paper copies of the survey to stewardship groups for which the research team can only get mail but not email addresses; because email contact is so common, we expect this to apply only to a small portion of the groups on our master list (for burden calculations, we are estimating 10% of groups will receive paper copies; in reality, we expect this to be less than 10%). Anyone who receives a paper version of the survey will be invited to complete the paper or online versions. Mailed surveys will follow the protocol recommended by Dillman, Smyth, and Christian (2009) – such as pre-notification letter, first survey questionnaire, thank you postcard, replacement questionnaire, and final reminder. Alternatively, for the stewardship groups without email addresses, we will contact them by phone as a substitute for mailings. Any groups reached by phone will be offered the options to complete the survey by phone, online, or mail-back paper version.

Response Rates

Phase 1. Response rates in Phase 1 (census) will be 80% or higher. This phase of the project is reaching out to project partners and other larger organizations that work with the project partners. In our experience, non-response in this phase is rare and is limited to contacts with severe time constraints.

Phase 2. Because the respondent population are NGOs and community based organizations, a response group known to have some of the lowest response rates (Hager et al.), we expect a response rate for the survey in Phase 2 to be approximately 50% in each location (overall n will vary depending on the universe determined in Phase 1).This is a minimally acceptable response rate according to Babbie as cited in Groves (2006). We accept this low response rate because:

• these data will be only used for descriptive purposes in reports and publications;

• these data will not be influential or foundational to any new federal regulations or policy;

• we accept that nonprofit organizations and community based organizations are known to be historically unlikely to complete surveys from federal agencies; and

• most surveys of this type and design often have very low response rates, as low as 10% in some cases but typically average in the 40% range (Hager et al.).

We believe that offering an incentive could increase our response rate by 10%. Singer & Ye (2013) discussed the effect incentives have in boosting response rates. We also believe that because the stewardship communities are so interconnected, there will be a high interest in promoting themselves along with others in the community to bring awareness to their efforts via STEW-MAP.

In instances of low response rates, if STEW-MAP partners in a given location are able to provide funds to support incentives, a lottery will be added to the survey process. Lotteries are a common method of providing incentives in web-based surveys. They have several benefits: they are easy to implement and have been shown to be effective in boosting response rates (Singer & Ye); by pooling resources for incentives, a more meaningful incentive can be offered; and they are easily offered to the entire respondent pool, even if added later in the data collection process. This approach meets guidelines in OMB’s survey guide of 2006: it will help improve data quality, is based on past experience, and will improve coverage of minority populations. Incentives offered will be targeted to stewardship efforts, such as a gift card for a home and garden improvement store or in-kind gifts such as passes to a local botanic garden.

Phase 3. In Phase 3, the optional follow up interviews, we expect a response rate of 80% or higher. The respondents will be one representative per organization from stewardship groups that are found to be primary connecting organizations in the network, based on network measures of centrality and betweenness. This characteristic – connecting groups – raises the likelihood that they will be willing to participate in the interviews.

Addressing Potential Non-Response Bias

While STEW-MAP data will not be generalized to the general population or to all stewardship groups in a city or region, nonresponse bias must still be evaluated. A low response rate does not mean that the results are skewed, but checks for such bias must be conducted. In a household survey of individuals, one method to check for nonresponse bias is to compare the demographics of respondents to those of the entire population being surveyed. If there is a significant difference on some characteristic (e.g., fewer low income respondents than in the population) then further investigation is needed to see if this difference affects the results.

STEW-MAP investigates groups and organizations, not individuals, and it also produces geographic data. Therefore, the investigations into potential response bias need to take these factors into account. To investigate for bias, we will:

1. Assess the geographic distribution of stewardship territories reported. Investigate areas with little reported stewardship via web searches, compared to the Phase 1 census information, and through knowledge of local STEW-MAP partners, and via GIS review (e.g., finding evidence of community gardens in Google Earth).

2. Compare survey response rates to Phase 1 census data by category of groups, including whether stewardship is the primary or secondary focus of the group, type of stewardship activities, race/ethnic and/or economic background of the stewards, and scale of stewardship activities (neighborhood to regional).

3. Compare first wave respondents to later respondents on types of stewardship they do, site types that they work on, primary focus, number of staff, % stewardship, etc. While this is a weak test of nonresponse bias, it may point out issues to investigate further through other means.

In each case, outreach and follow up survey and/or interviews will be conducted to determine if there is a bias in the data gathered with regard to the types of groups conducting stewardship, their goals, and other primary data assessed in Phase 2. When necessary, an abbreviated (10-minute version) survey will be offered; this version will include only the “core” questions about the organization’s structure and stewardship work without the social network questions or mapping section. People who complete the 10-minute version will be counted as respondents.

We will also conduct a brief non-response bias survey with 10% of all non-respondents (n=500). We estimate a 10% response rate (n=50). In many instances, non-respondents do not even wish to participate in an abbreviated survey, and our efforts to identify bias will be based on identifying who our data does not represent.

The responses from the non-respondent survey will be compared to the responses from all respondents returning the full version of the survey. We will analyze the results and any evidence of non-response bias will be examined and the implication will be discussed in any reports of publication that uses this data.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separate­ly or in combination with the main collection of information.

Pilot Testing

The STEW-MAP survey was pre-tested in multiple cities to receive feedback on questions and to determine the level of usable responses. Items in some of the survey questions will be edited to be relevant locally. For example, the question asking stewardship groups to indicate the types of sites they work on would require alternating landscape or ecosystems types for a more relevant response. For example, “prairie” will be changed to “mountain trail” if the survey is conducted in Colorado versus Chicago. This feedback will be solicited from local STEW-MAP partner organizations in each city or region where a STEW-MAP project is conducted. Likewise, based on the pretest we were advised that some survey questions may need to be eliminated because of limited local relevance.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

This project and its methodology have been reviewed by the National Agricultural Statistics Service.

In addition, the following people were consulted about statistical design and analysis of previous STEW-MAP projects:

• Dana Fisher, PhD, Associate Professor, University of Maryland Department of Sociology.
  
  

• Jarlath O’Neil-Dunne, Director of the University of Vermont's Spatial Analysis Laboratory.

Literature cited:

Dillman, D.A.; Smyth, J.D.; Christian, L.M. (2009). Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. Hoboken, NJ: Wiley and Sons.

Groves 2006. Nonresponse Rates and Nonresponse Bias in Household Surveys. Public Opinion Quarterly. V70, n5 pp 646-675.

Hager et al. 2003. "Response rates for mail surveys of nonprofit organizations: A review and empirical test". Nonprofit and Voluntary Sector Quarterly, vol 32, no 2).

Singer and Ye. 2013. The use and Effects of Incentives in Surveys. Annals of the American Academy, 645, January 2013.

Page 1 of 6