ATTACHMENT 2: Approval Form for DOI Programmatic Clearance for Customer Satisfaction Surveys (OMB Control Number 1040-0001).
PPA Tracking Number:_____CSS-3_________________
1. Survey Title: Assessing Customer Satisfaction with Riparian Training and Assistance Offered by the Interagency National Riparian Service Team and State Riparian Teams as Part of the Creeks and Communities Strategy
2. Bureau: Bureau of Land Management (BLM)
3. Abstract: The BLM requests approval to collect customer satisfaction data regarding the delivery, quality, value and usefulness of the products, information and services provided by DOI Personnel, Contractors and Partners in the Creeks and Communities Strategy. This effort was initiated in 1996 by BLM and the Forest Service (FS), in partnership with the Natural Resources Conservation Service (NRCS), to accelerate cooperative riparian restoration and management in the Western United States. The interagency and interdisciplinary National Riparian Service Team and State Riparian Teams serve as the implementation arm of this effort focusing on training and problem solving assistance. The NRST has contracted with Oregon State University to develop and administer this customer information collection. Data obtained from these customer interactions will be used to identify opportunities for strategy improvement. Mail-back surveys and in-person interviews will be used to collect valuable customer information that will assist the BLM in complying with the Administration’s guidance.
4. Bureau/Office Point of Contact Information: Laura Van Riper, PhD. Social Scientist. Bureau of Land Management, National Riparian Service Team. 3050 NE Third St. Prineville, OR 97754. Phone: 541-416-6702 Fax: 541-416-6798 Email: [email protected]
5. Principal Investigator Information:
(Principal Investigator) Hannah Gosnell, PhD. Assistant Professor. Department of Geosciences, Oregon State University. Wilkinson 104 (Office Wilkinson 202). Corvallis, OR 97331-5506. Phone: 541-737-1222 Fax: 541-737-1200 Email: [email protected]
(Co-Investigator) Lena Etuk. Social Demographer. Extension Family & Community Health, Oregon State University. 161 Milam Hall. Corvallis, OR 97331 Phone: 541-737-6121. Fax: 541-737-0999. Email: [email protected]
6. Name of Program or Office Conducting the Survey: Bureau of Land Management, National Riparian Service Team has contracted with Hannah Gosnell and Lena Etuk to develop, conduct and analyze the survey and interviews.
7. Description of Customers/Services Provided: The NRST and State Riparian Teams provide training and problem solving assistance in riparian assessment and management across the western United States. Customers include Federal land management and regulatory agencies, state agencies, county government, Tribal government, university affiliates, non-government organizations, and private individuals with an interest in riparian restoration and management.
8. Survey Dates: Interviews will occur between June and December 2010. Surveys will be administered between August and December 2010.
9. Type of Information Collection Instrument: Mail-back survey and individual, in-person and telephone interviews.
10. Survey Development: Lena Etuk assisted with the development of mail-back survey content and statistical analysis (Title: National Riparian Service Team (NRST) and State Riparian Teams Proper Functioning Condition (PFC) Riparian Assessment Training, Grazing Management for Riparian Areas Training and Site-Specific Riparian Assistance Past Participant Survey). Hannah Gosnell assisted with the development of the interview content and data analysis (Title: Interview Guide for National Riparian Service Team (NRST) Site-Specific Riparian Assistance Participants).
Over the past year, extensive interaction between Hannah and Lena and all NRST members occurred to help identify and frame research questions. All NRST members and some State Riparian Team members reviewed both draft instruments to be sure that the questions accurately and clearly reflected the purpose and objectives of the various activities. Furthermore, both instruments were pre-tested (see below). Combined, the survey and interview guide address the following topic areas: (1) delivery, quality and value of the products, information and services; (2) interactions with DOI personnel and contractors; and (3) general demographics.
In order to increase the validity of the study, multiple methodological approaches will be used to balance out the strengths and weaknesses of different approaches (McCool and Guthrie 1998, Moss 1992, Guba & Lincoln 1989, Patton 1980). First, a large scale mail-back survey will be used to gather information that is can be generalized across the entire population of individuals who participated in trainings and assistance offered as part of the Creeks and Communities Strategy between 2003 and 2010. Random, unequal probability sampling stratified by activity type (PFC training, grazing training, and NRST assistance) will be used to draw a sample of past participants.
Second, an in-depth analysis of eight locations where NRST site-specific riparian assistance was offered during 2003 and 2010 will be conducted using a case study approach, including individual in-person and telephone interviews. Due to the uniqueness and complexity of each site-specific assistance1, this portion of the study will provide an opportunity for researchers to focus in on the nuances of each assistance effort and gain a more complete understanding of past participant satisfaction and the usefulness of these activities. The goal of sampling in this stage is to provide a detailed description of the range of past NRST site-specific riparian assistance participant experiences, rather than to determine the extent to which different types of experiences are distributed across the population (Bellah et al 1985). Therefore, purposive sampling principles rather than random sampling principles rather than random sampling will be used. The results generated from the case studies will complement the survey data by providing a more complete understanding (thick description) of the site-specific assistance aspect of the Creeks and Communities Strategy. With thick descriptions, the context and meanings of experiences, events and scenes can be presented in detail by those involved in it (Geertz 1976). Furthermore, the use of an open-ended, semi-structured interview process allows for the capture of unanticipated themes or ideas that may emerge. Finally, this type of interview process also provides the researcher with an opportunity to clarify (or probe) the meaning of individual responses.
Survey (Questionnaire) Instrument Development Methodology
After developing the survey instrument, one OSU faculty member (Lena Etuk) conducted cognitive pretests of the survey instrument with past participants of PFC training, Grazing training, or Site-Specific Riparian Assistance. Eighteen individuals were randomly selected as potential participants in the pretest; six in each of the three training and assistance categories. Etuk attempted contact with all 18 via email and phone, with multiple follow-ups. Contact was actually made with 13 individuals in the form of voicemails, successfully delivered emails, or direct telephone conversations. Two individuals refused to participate in the pretest, due to time constraints. Seven individuals agreed to participate in the pretesting, but two cancelled at the last minute. A total of five individuals participated in the pretest of the survey instrument:
Four were women
One was a past Grazing training participant (2003)
One was a past PFC training participant (2004)
Three were past Service Trip participants (2003, 2006, 2007)
Two worked for state agencies at the time of the training or Service Trip
One worked for the Bureau of Land Management at the time of the training or Service Trip
One worked as a rancher at the time of the training or Service Trip
One worked as a member of a natural resource conservation nonprofit at the time of the training or Service Trip
Before the pretests were conducted, each participant received a hard copy of the survey in the mail from OSU. Four of the five individuals, who completed the pretest, were instructed not to complete the survey until they were called to participate in the pretesting, and one was instructed to complete the survey before the phone call and to note the amount of time it took to fill out the survey, and any areas of difficulty. At the time of the pretest interview, two of the five had completed the survey beforehand.
In each of the five pretest interviews, Etuk instructed participants to fill out sets of questions (corresponding to a section of the survey or a sub-set of questions in a section) in silence or aloud and then respond to follow-up questions. Follow-up questions included:
“How did you answer question X?”
“Why did you answer question X that way?”
“In question X, what did you think was meant by the phrase/term…?”
“For question X, list some examples of things you associate with the phrase/term…”
“Describe a scenario that you were thinking of as you answered question X.”
“How did you know that your NRST experience had an influence on you in this way?”
Responses to follow up questions and the survey itself were recorded by Etuk and assessed for themes at the conclusion of pretesting. Overall, pretesting revealed that the survey overall was a comprehensive, valid, and reliable tool with which to assess participants’ experiences with the NRST and State Riparian Teams, with questions that were not overly taxing to their memories. Despite this positive assessment, the pretesting also revealed some areas in which the instrument had to be improved. These improvements included the following:
Simplifying and reducing wording , removing redundant or confusing survey items, and clarifying the meaning of terms;
Reformatting the questionnaire to improve readability, including changing the order and placement of questions in order to clarify their relationship with other questions;
Clarifying the elements of the NRST and State Team experience that the respondent should be considering as he or she answers the questions (i.e., consider all of the trainings and assistance he or she participated in versus just one); and
Focusing the respondent on a particular set of experiences they may have had after the training or assistance.
Case Study (Interview) Instrument Development Methodology
After developing the interview guide, OSU faculty member (Hannah Gosnell) and research assistant (Robyn Paulekas) pre-tested the instrument using the Sprague River Basin (Beatty, Oregon) as a pilot-study. Using the draft interview guide, nine in-person interviews were conducted with two Federal employees and seven private individuals. Overall, the pre-test indicated that the interview guide was a comprehensive, valid and reliable tool with which to assess participants’ experiences with the site-specific riparian assistance offered by the NRST. In an effort to further improve the instrument, a few questions were modified to enhance clarity.
11. Survey Methodology:
Survey (Questionnaire) Methodology
Respondent universe: Adults who participated in any one of three types of workshops offered by the NRST and State Riparian Teams between 2003 and 2010 (PFC training, grazing management training and NRST assistance).
Sampling plan/procedures: The frame that will be used to draw a sample of past participants to receive the survey consists of the combined registration forms for individuals who participated in each of the three workshops offered by the NRST and State Riparian Teams between 2003 and 2010. The completeness of the sampling frame is constrained by the following: not all individuals signed or provided contact information on the registration form for PFC training, grazing training or NRST assistance; some completed the registration form but indicated a desire to not be contacted; and in some instances, participant registration forms were either not provided at the session or were subsequently lost.
In order to draw a sample of past workshop participants to receive the survey we will use random, unequal probability sampling stratified by workshop type. For each set of participants in each workshop type (PFC training, grazing training, and NRST assistance) we will randomly draw a set number of past participants, which corresponds to the total number of contacts needed to be 95% confident that the population value falls within + or – 6% of our estimates, given an anticipated response rate of 40%.
Instrument administration: Survey administration will follow the Dillman (2007) method.
Once the individuals have been selected, a pre-notice letter will be sent out via mail informing them of their selection for the survey (attachment 1).
One week later, survey packets will be mailed to these sampled individuals. The packets will contain the following:
Cover letter explaining the survey (attachment 2)
An informed consent letter, indicating that the submission of a completed survey is indication of his or her agreement to participate (attachment 3)
A stamped, pre-addressed return envelope for the completed survey
The questionnaire (attachment 4)
One week after the survey has been delivered to a sampled individual, a follow-up thank you/reminder postcard will be mailed (attachment 5).
Two weeks after the initial survey packet has been delivered to a sampled individual, the ID database of all sampled individuals will be matched to the IDs of returned surveys. Individuals who returned surveys will be deleted from the database of mailing addresses. Those remaining in the database will receive another follow-up packet, three weeks after receiving the initial survey packet. This packet will include the following
A second reminder letter (attachment 6)
A replacement copy of the survey (attachment 4)
An informed consent letter, indicating that the submission of a completed survey is indication of his or her agreement to participate (attachment 3)
A stamped, pre-addressed return envelope for the completed survey
OSU employees will enter the information from completed questionnaires into a database. Each individual survey will be assigned an identification code, which will be completely independent of the individual respondents’ identity. The raw data, minus any identifiers will be provided to the NRST, along with a summary report of findings.
Expected response rate/confidence levels: This survey does not fit the criteria of ‘influential’ information collection as defined by OMB (i.e., does not have a substantial impact on important public policies or on private sector decisions). Rather, the information will be used to improve program implementation. As a result, efforts will be focused on achieving a 40% response rate. Additionally, we anticipate receiving enough completed surveys from each workshop population to be 95% confident that the true population values are within 6% of the sample estimates we obtain.
Strategies for dealing with potential non-response bias: A similar survey of this population, conducted by Van Riper (2003), resulted in a 31% response rate. 50 follow-up phone calls to non-responders revealed that 55% of those who did not respond were individuals who had moved or were retired, did not receive the survey, or did not participate in the course. These reasons for non-response are not likely to be associated with our key outcomes of satisfaction with the training or assistance, knowledge gain, or behavior change.
Since the composition of the population that will be sampled in 2010 is not different from the 2003 population we can use the 2003 findings related to non-response assessment to estimate the 2010 non-response bias. From the 2003 assessment we conclude that non-response was driven by inaccuracies in the sampling frame; factors that are not associated with outcomes. As a result, there are no plans to complete a non-response bias check at this time for the 2010 survey.
Description of any pre-testing and peer review of the methods and/or instruments (recommended): See question 10.
Communicating PRA compliance information: PRA compliance language will be inserted into the first page of the survey instrument, along with the OMB control number.
Case Study (Interview) Methodology
Respondent universe: Adults who participated in NRST site-specific riparian assistance trips between 2003 and 2010. Individual participants who completed registration forms, but indicated a desire to not be contacted in the future have been removed from the sampling frame.
Sampling plan/procedures: We will use an “expert choice” approach to purposive sampling to identify potential interviewees, relying on the judgment of NRST members who facilitated the site-specific riparian assistance in each study area. This approach stems from the idea that the qualitative portion of the study is one of "discovery" in contrast to the survey component, which is more about testing hypotheses using probabilistic sampling. In keeping with this method, we will specifically choose potential interviewees with certain characteristics that make them representative of the population that participated in a given assistance. Admittedly, the inability to determine the probability of selection associated with the selected interviewees can result in some bias, but the NRST has deemed it important to gather, in addition to quantitative data, more in-depth feedback from “experts” representing specific types of participants, e.g. employees of each of the federal agencies involved, private landowners engaged in grazing, recreationists, etc. An expert choice approach allows the researcher to concentrate not just on “typical” participants, but on participants displaying wide variety, including extreme cases (e.g. participants who are known to have been dissatisfied with the assistance delivery). In this sense, a non-probability approach may be informative in a way that conventional probability sampling cannot be (Denscombe 2007).
Instrument administration: Interviews will take place both in-person and over the telephone.
Once the individuals have been selected, a pre-notice letter will be sent out via mail informing them of their selection for the interview (attachment 7).
Individuals who agree to participate in telephone interviews will receive in the mail an informed consent form (attachment 8), with a request to sign and return ; the PRA compliance form (attachment 10), with a request to review; and a self-addressed stamped envelope to facilitate return of the consent form. Those who agree to participate in an in-person interview will receive this information on site.
The consent form provides an option for interviewees to participate without being audio recorded, in which case the interviewer will take notes.
Each interview will be conducted wherever the interviewee feels most comfortable. To ensure the interviewee’s privacy is protected, researchers will suggest meeting interviewees at a place of their choosing. Each interview is designed to last one hour, but interviewees will have the freedom to determine the amount of time they devote to being interviewed. Interviews will be semi-structured, guided by an interview guide (Patterson et al 1998, Charmaz 1991, Kvale 1983) that identifies themes to be addressed in the interviews and a series of possible probes for each theme to ensure that interviews are systematic and focused enough to cover relevant and comparable (across interviews) information while providing the flexibility needed to be responsive to and explore emergent data and gain clarity (attachment 9). Within one month of each completed interview, respondents will receive a thank-you letter in the mail (attachment 11).
Unless the interviewee has opted not to be recorded, interviews will be recorded using a digital recorder and the appropriate software to download the recordings into a file on a password-secured computer in the PI’s office in the Geosciences Department. If the interviewee opts not to be recorded, notes will be taken. Recordings and notes will be kept for the duration of the study and up to 18 months afterwards on a password-secured computer in the PI’s office in the Geosciences Department, after which time they will be discarded. All names and identifying features from all interview data and write-up material will be removed; numbers will be substituted for interviewee names. The audio-recordings will be transcribed and coded. Hard copies and digital copies of these transcriptions will be kept for the duration of the study (stored in a locked file cabinet or on a password-secured computer in the PI’s office in the Geosciences Department) and up to 5 years afterwards, after which time they will be disposed of in a manner which maintains the confidentiality of the participants. Code lists will be stored in a separate secure location in the Geosciences Department. Data analysis will begin as soon as interviews are transcribed. Interviews will be analyzed using NVivo software.
Upon completion of this study, the NRST will receive a summary report of study findings. This information will be presented in aggregate form and will not contain any identifiers. The NRST may also request copies of portions of the written interview transcripts minus any identifiers.
Expected response rate/confidence levels: Based on experience in the Sprague pilot study, a 75% response rate is anticipated. In the Sprague, twelve people were contacted but only nine were interviewed. Three people either refused or did not respond to the request for an interview.
Strategies for dealing with potential non-response bias: n/a
Description of any pre-testing and peer review of the methods and/or instruments (recommended): See question 10.
Communicating PRA compliance information: A form with required PRA compliance language and the OMB control number will be handed to each respondent at the outset of each interview (attachment 11)
12. Total Number of Contacts/Expected Number of Respondents:
Survey (Questionnaire)
The following numbers represent estimates, because all of the registration forms have not yet been compiled and the sample will include 2010 participants whose actual numbers are unknown at this time. As previously stated, the sample will be stratified across the 3 different NRST and State Team offerings and the following number of sample draws will be made. The number of sample draws was calculated based on an estimated 40% response rate and a desire on the part of the research team to be 95% confident that the population values are within 6% of our sample estimates, for each workshop type:
PFC Trainings: Total Population = 1,000-2,000 participants
Number of Contacts = 550 participants
Number of Respondents = 220 participants
Grazing Trainings: Total Population = 450-550 participants
Number of Contacts = 450 participants
Number of Respondents = 180 participants
NRST Assistance: Total Population = 500-1000 participants
Number of Contacts = 475 participants
Number of Respondents = 200 participants
Total Number of Contacts = 1475
Total Number of Respondents = 600
Case Study (Interview)
As previously stated, the sample will be compiled using an expert choice sample . There will be four on-site cases selected where ten, one hour, in-person interviews will be conducted for a total of forty (40) respondents. There will be an additional four (4) cases selected where five, one hour, telephone interviews will be conducted for a total of twenty (20) respondents. In all, the case study section of this effort will result in feedback from sixty (60) respondents.
13. Estimated Time to Complete Initial Contact/Instrument:
Survey (Questionnaire)
The estimated time to read the pre-notice letter, the consent form and the cover letter is 10 minutes. The estimated time to complete the survey is 30 minutes.
Case Study (Interview)
The estimated time per initial contact (for people to read the recruitment letter and indicate their willingness to participate when contacted by telephone) is 5 minutes. The estimated time per second contact (schedule an interview, read/sign/return mail the consent form and read the PRA compliance form – once agreed to participate) is 10 minutes. The estimated time per response is one hour per interview.
14. Total Burden Hours:
Survey (Questionnaire)
(1475 total number of contacts x 10 minutes/contact) + (600 total number of respondents x 30 minutes/response) = 14,750 minutes + 18,000 minutes = 32750 minutes/60 minutes = approximately 546 hours
Case Study (Interviews)
(80 total number of contacts x 5 minutes/initial contact) + (60 total number of contacts x 10 minutes/second contact) + (60 total number of interviews x 60 minutes/interview) = 400 + 600 + 3600 minutes = 4600 minutes/60 minutes = approximately 77 hours
Total Burden Hours for Study
546 hours (questionnaire) + 77 hours (interviews) = 623 hours
15. Reporting Plan:
Survey and interview data will be analyzed in such a way as to answer the research questions guiding this study effort. OSU faculty will write a report that covers both the descriptive disposition of the survey sample as well as the relevant findings from the survey that relate to the overarching research questions. A similar effort will be undertaken on the interview data. The survey findings will then be combined with findings from the interviews in order to gain more complete insight into the research questions.
A full report will be provided to the National Riparian Service Team, as well as archived with the Office of Policy Analysis (PPA). In addition, OSU may do some analysis on the data at the individual level, for additional research reports, papers, or presentations. In that event, no identifying information will be revealed, as findings will be reported in the aggregate.
16. Justification, Purpose and Use:
Survey Justification and Purpose: The purpose of this study is to collect customer satisfaction data regarding the delivery, quality, value and usefulness of the products, information and services provided by DOI Personnel, Contractors and Partners in the Creeks and Communities Strategy. Data obtained from these customer interactions will be used to identify opportunities for strategy improvement and will assist the BLM in complying with the Administration’s guidance.
Survey Goals: The study is designed to gain insight into participant satisfaction with the NRST and State Riparian Team sponsored training and assistance, as well as to ascertain the usefulness of these activities. Specifically, the following research questions were identified:
Are respondents satisfied with the NRST/State Riparian Team trainings or site-specific assistance?
How does participation in a training (PFC or Grazing) or site-specific assistance contribute to individual knowledge gain and behavior change across the three dimensions (science, collaboration, and management change)?
How does participation in a training (PFC or Grazing) or site-specific assistance contribute to riparian management and monitoring change?
How does a community’s experience or participation in a training (PFC or Grazing) or site-specific assistance contribute to group behavior change across the three dimensions (science, collaboration, and management change)?
Who participates in the trainings (PFC or Grazing) and site-specific assistance?
What factors impede changes in individual behavior, and changes to riparian condition?
Does participation in one offering of the strategy encourage/lead to participation in other offerings of the strategy?
Have individuals or groups experienced any impact from participation in the strategy above and beyond the intended outcomes?
Utility to Managers: The results of this study will be useful to managers insofar as they will provide insights into the value and usefulness of the Creeks and Communities Strategy. In addition to identifying opportunities for strategy improvement, managers will also be able to get of sense of what is working well and may choose to extrapolate those principles and practices to training and assistance related to other resource areas.
How will the results of the survey be analyzed and used:
Results of the survey and the interviews will be analyzed by OSU in such a way as to answer the research questions outlined in question #16b. The study will be used internally by the National Riparian Service Team to advocate for continued funding and support for the Creeks and Communities Strategy, to make programmatic changes as appropriate, to strengthen those program areas that are working well and to contribute to existing discourse around cooperative riparian restoration and management. The raw data files may be used internally by the National Riparian Service Team to further assess program effectiveness as appropriate and to address future data needs that may arise. The study may be used by OSU faculty members to inform scientific discourse on collaborative natural resource management through scholarly presentations, journal articles, book chapters, or other professional venues.
How will the data be tabulated? What Statistical Techniques will be used to generalize the results to the entire customer population? How will limitations on use of data be handled? If the survey results in a lower than anticipated response rate, how will you address this when reporting the results?
Survey (Questionnaire)
Quantitative survey data will be analyzed using STATA statistical software. The methods that will be used to analyze the data are largely exploratory in nature. After descriptive statistics are calculated and considered, however, tests of significant differences across sub-populations may be appropriate (Chi-squared, t-tests, Wald statistics, or others). It may also be appropriate to examine the association among variables. If this is the case, Ordinary Least Squares regression or Maximum Likelihood Estimation will be used to better understand the relationship of key variables, holding other factors constant.
In all of these scenarios, data analysis will reflect the survey sampling design in order to facilitate the generalizability of the data to the customer population. Given the intent to use these sample data as estimates of the total population, it is necessary to use statistical weighting procedures that reflect the way the sample was drawn and correct for non-response of individuals.
Sampling weights can account for the under-representation of certain elements of the population due to errors in the sample. We will construct and apply weights to individual respondents to make each represent a targeted number of NRST/State Riparian Team workshop participants, thus bringing the sample closer in line with the true population.
The first step of the weighting procedure will account for the sampling design. Each individual record will be assigned a base weight that corresponds to the probability of the individual being selected. In addition, we will adjust the sample for non-response. Assuming that respondents are similar to non-respondents makes it possible to assign an additional weight (a weight class adjustment) to each individual respondent to make them represent non-respondents from the same stratum. These two weights will be combined for each respondent to produce the final weight. These weights will be applied to the data using STATA statistical software.
If the survey results in a lower than 40% response rate, then a sample of non-respondents will telephoned in order to collect only enough information to assess whether the respondents and non-respondents differ to a significant degree on basic demographic factors and workshop attendance patterns.
If we have reason to believe that the sample suffers from non-response bias, then we will not implement the weight class adjustment to the sample. In addition, we will clearly explain this limitation in any and all reports.
Case Study (Interview)
The qualitative component of the study will not involve statistical analysis, nor will there be an expectation that our results can be generalized to the entire customer population. Rather, the analysis of the case study data will be inductive in nature, and a non-mathematical process of interpretation will be used to document the nuances of the settings under study in an effort to gain novel understandings. We expect the data collected through the semi-structured interviews to complement the quantitative data collected through the surveys. While survey data is useful for identifying general trends, in-depth qualitative analysis will be extremely useful for understanding the nuances of why certain aspects of the NRST program are working, and others are not. Interviewees can provide “thick” description of their experiences, and their perceptions about the ways in which they and their community have been affected (or not) by a given program. If the goal of the NRST is to improve delivery of its programs, this kind of analysis is essential for devising new strategies.
In terms of analytical methods, interviews will be recorded and transcribed and detailed field notes will be written following each interview. Analysis will involve reading through the interview transcripts and field notes, identifying recurring themes, and developing an organizing system using NVivo software (qualitative analysis software). The purpose of an organizing system (Patterson et al 1998, Tesch 1990) is to identify predominant themes through which interviews can be meaningfully organized, interpreted and presented. The process of developing an organizing system is the ‘analysis,’ while the final organizing system is the product of the analysis. The development of an organizing system is a systematic process beginning with the identification of meaning units (codes), followed by the identification of themes, and ending with an analysis of the interrelationships among these themes. These insights will be used to inform a set of findings regarding past customer satisfaction and recommendations for improvement in the future.
Is this survey intended to measure a Government Performance and Results Act (GPRA) performance measure? If so, please include an excerpt from the appropriate document. n/a
17. Literature Cited:
Bellah, R. N.; Madison, R.; Sullivan, W. M.; Swidler, A.; and Tipton, S. M. 1985. Habits of the heart: individualism and commitment in American life. Berkley: University of California Press.
Denscombe, M. 2007. The Good Research Guide: For Small-Scale Social Research Projects, 3rd ed. Berkshire, UK: Open University Press.
Dillman, Don. (2007). Mail and Internet Surveys: The Tailed Design Method. 2nd Edition. Hoboken, NJ: John Wiley & Sons, Inc.
Charmaz, K. 1991. Translating graduate qualitative methods into undergraduate teaching: intensive interviewing as a case example. Teaching Sociology 19:384-395.
Geertz, Clifford. 1973. Thick description: toward an interpretive theory of culture. In The Interpretation of Cultures: Selected Essays. Ed. By Clifford Geertz. New York: Basic Books, Inc.
Guba, Egon G. and Lincoln, Yvonna S. 1989. Fourth Generation Evaluation. Newbury Park, CA: SAGE Publications.
Kvale, S. 1983. The qualitative research interview: a phenomenological and hermeneutical mode of understanding. Journal of Phenomenological Psychology 14:171-196.
McCool, S. F. and Gutherie, K. 1998. Building consensus: legitimate hope or seductive paradox. Report submitted to: Bitterroot Ecosystem Management and Research Project. USDA Forest Service. Rocky Mountain Research Station.
Moss, P. A. 1992. Shifting conceptions of validity in educational measurement: implications for performance assessment. Review of Education Research 62:229-258.
Patterson, M.E.; Watson, A.E.; Williams, D. R.; and Roggenbuck, J. R. 1998. A hermeneutic approach to studying the nature of wilderness experiences. Journal of Leisure Research 30:423-452.
Patton, Michael Q. 1980. Qualitative Evaluation Methods. Beverly Hills, CA: SAGE Publications.
Tesch, R. 1990. Qualitative research: analysis types and software tools. New York: Falmer Press.
Van Riper, L. 2003. Can agency-led initiatives conform to collaborative principles? Evaluating and reshaping an interagency program through participatory research. Ph.D. dissertation. University of Montana, Missoula, MT. 321 p.
1 Unlike the PFC and Grazing trainings, which follow a similar format at all locations; each site-specific assistance is crafted to fit the particular social and ecological characteristics found in each area.
| File Type | application/msword |
| Author | lvanripe |
| Last Modified By | djbieniewicz |
| File Modified | 2010-04-29 |
| File Created | 2010-04-29 |