CSA QUALITATIVE RESEARCH PROTOCOL
Public Assistance (PA) Surveys
The Public Assistance Program is constantly striving to streamline their process. To accomplish this, there are often programmatic changes and new regulations that aren’t captured in the customer satisfaction surveys. Periodic qualitative research is essential to measure satisfaction with these changes and give Public Assistance more immediate feedback. Additionally, focus groups/interviews can also assist with the re-design of survey questionnaires.
This protocol was developed to provide consistent guidelines within the Customer Survey and Analysis (CSA) Section of FEMA who plan to conduct qualitative studies for issues related to the Public Assistance (PA) surveys which include: Public Assistance Initial (PAI) and Public Assistance Assessment (PAA).
Qualitative research may include focus groups (see Appendix A) or interviews (see Appendix B).
There are three phases to conducting qualitative research and these phases are explained below:
PHASE 1: BEFORE THE FOCUS GROUP/INTERVIEWS
Meet with management/leadership to identify overall purpose and outcomes for the focus group.
Discuss possible hypotheses.
Identify funding sources.
Determine target population and number of sessions needed.
The target population with always be Public Assistance applicants. Disasters will be chosen for sample selection based on the research question.
Example: If Public Assistance is interested in gathering immediate feedback on state-led disasters, we select state-led disasters as our target population.
Sample selection is always partially restricted by funding- we typically only have funding to conduct qualitative research in a couple locations. The density of PA applicants in an area dictates the feasibility of focus groups (applicants come to central location) or interviews (interviewers travel to applicant).
Focus Groups: There is an annual allotment of 360 hours for focus groups. This includes 12 participants per session, with 3 sessions for each of the 10 FEMA regions. Each focus groups session lasts approximately 2 hours, with an additional hour allotted for travel.
Interviews: There is an annual allotment of 200 hours. This includes 2 participants per 1-hour interview, with 10 interviews for each of the 10 FEMA regions. Travel not required.
Identify participants:
Focus groups typically have between six and twelve participants. Fewer than six participants may limit the conversation and yield poor data while more than twelve can be unmanageable. We allow for maximum of 12 per session with the anticipation that some may not attend.
Interviews typically work best with 1-2 people, although be aware that more people may show up.
Develop a list of key attributes to seek in participants based on identified purpose.
Secure names and contact information and send invitations.
Generate questions.
Based on the purpose and goals of the qualitative research, identify no more than 5-6 questions.
Revisit the questions to make sure they will promote appropriate discussion to address research questions.
Order the questions from general to specific.
Scope of questions may include topics related to the following:
Satisfaction with PA New Delivery Model
Feedback on 428 vs. 408 (receiving fixed estimates or estimates based on costs)
Feedback on state-led disasters
Usability of the Grants Portal
Recent programmatic changes not captured in surveys
Long-term recovery
6. Select a facilitator and note-taker
The facilitator should not be someone who directly oversees the issue being researched. This may make participants less open to sharing their thoughts or concerns.
The facilitator should be able to keep the discussion going, deal tactfully with difficult or outspoken group members, and make sure all participants are heard.
The facilitator should ask the questions and probes, but not participate in the dialogue or correct participants.
If possible, a program analyst can sit in for specific program questions.
7. Choose the location
For focus groups, choose a location which is comfortable, easily accessible, and where participants can see one another.
Choose a setting which does not bias the information gathered. Preferably libraries or other public forums with meeting rooms.
For interviews, the location is typically dictated by the interviewee and is usually at their place of work.
8. Develop script: Moderator’s Guide (See Appendix A for Focus Groups, Appendix B for Interviews)
Part one: Welcome participants and make introductions.
For Focus Groups: Provide logistics of building related to emergency exits and restrooms and sign consent forms.
Part two: More about FEMA CSA and reasons for research. Explain that information is confidential and no names will be used.
Part three: Ask for introductions from the group (focus group).
Part four: Ask prepared questions; remember to use probes and follow up questions to explore the key concepts more deeply.
Part five: Close the focus group/interview – thank participants, give them contact information for further follow up if requested, and explain how data will be analyzed and shared.
PHASE 2: CONDUCTING THE QUALITATIVE RESEARCH
1. Bring materials:
Notebook/computer
Flip chart paper/ no board is available (if available for Focus Groups)
List of participants
Script
Name tags
Location signage (focus group)
Watch or clock (focus group)
Consent forms
2. Arrive before the participants to set up room, display signs directing participants to room, etc. (focus group).
3. Introduce facilitator and the note-taker (if applicable) and carry on the focus group
according to the script.
4. Conduct the session, being mindful of the following:
Set a positive tone while empathizing with their situation.
Make sure everyone is heard; draw out quieter group members.
Probe for more complete answers.
Monitor questions and the time closely – it is the job of the facilitator job to make sure the group is on track.
Don’t argue a point with a participant, even if they are wrong. Address it later if needed.
Thank participants and let them know FEMA values their input considering the disruption in their lives when dealing with a disaster.
PHASE 3: INTERPRETING AND REPORTING THE RESULTS
1. Summarize each meeting
Immediately after the meeting, the facilitator should write up a quick summary of his/her impressions.
Transcribe notes. This should be done as soon as possible after the focus group/interview has been conducted.
If multiple facilitators have been used for two or more groups, discuss impressions with the other facilitators before reviewing the transcript.
2. Analyze the summaries
Read the notes and look for themes/trends. Write down any themes which occur more than once.
Context and tone are just as important as words. If comments are phrased negatively or triggered an emotional response, this should be noted in the analysis.
Interpret the results
What are the major findings?
What would be possible recommendations?
3. Write the report
The report should include the research purpose, outcomes, process, findings (major themes and minor themes), and recommendations.
Submit the report to management.
4. Make adjustments/take action on what was learned (After-action review)
Schedule a meeting with management and/or focus group facilitators to discuss the implications.
Highlight the main themes, and/or issues that arose.
These themes/issues can be related to logistics of the research or program discussion.
Prioritize the results and make action plans for the most important priorities.
FOCUS GROUP MODERATOR GUIDE:
PUBLIC ASSISTANCE
CUSTOMER SATISFACTION
ANALYSIS SECTION
PUBLIC ASSISTANCE CUSTOMER SATISFACTION FOCUS GROUPS Moderator Guide |
10 min
|
INTRODUCTION:
|
TRANSITION: Are there any questions before we get started? |
10 Min |
FAVORITE COMPANIES:
1. LIST FAVORITE COMPANIES: Studies have shown that it is important for customers to feel good about the companies they do business with. Let’s begin with finding out which companies are your personal favorites? These would be companies that impress you or make you feel good about doing business with them.
For this exercise I would like each of you to come up with at least one company that is a favorite and has made a positive impression on you. Take a few minutes to think about your favorite companies and then I will ask each of you to tell me the name of one or more of your favorite companies.
|
20 Min |
ATTRIBUTE SERIES:
2. LIST CUSTOMER SERVICE ATTRIBUTES: Next, I would like you to tell me specifically what these companies do to distinguish them apart from other companies and make you feel good about doing business with them.
Who would like to start?
|
Now I would like to shift your focus from your experience with your favorite companies to your recent experience with FEMA. |
5 Min |
3. CAPTURE ATTRIBUTES UNIQUE TO FEMA:: Considering all the important service attributes we’ve discussed, Which ones from your perspective would be important to you in conducting business with FEMA?
3a. ADDITIONAL FEMA ATTRIBUTES (NOT ON LIST) Are there any additional attributes that would be important in doing business with FEMA that are not on our list?
|
10 Min |
4. FIVE MOST IMPORTANT FEMA ATTRIBUTES: Ok, we have a good list here, now I would like you to take your pad and pen and over the next few minutes write down what you feel are the five most important customer service attributes in doing business with FEMA?
Ok, let us see which five you selected. (Begin Polling)
|
Now that we have your feedback regarding the customer service attributes that are the most important to you, I would like you to think back to the time immediately and the days after the disaster, before the Kickoff Meeting. |
|
5 min
|
Information: 5. LIST INITIAL INFORMATION WANETED/NEEDED What specifically did you expect FEMA would do to help your organization recover from the disaster?
|
5 Min |
5a. DIFFERENCE IN EXPECTATIONS Now, let’s talk about the expectations we’ve listed. Did any of you find that the assistance you received did not match your initial expectation and if so how was it different?
5b. POINT REALIZED EXPECTATIONS NOT MET At what point in the recovery process did you realize that that your expectations may not be met?
|
10 min.
|
PERSONAL INTERACTION/CUSTOMER SERVICE:
6. Understanding of Eligibility Requirements MoDid the field staff provide you with the eligibility requirements? Were requirements clear?
6a. SUGGESTIONS TO IMPROVE UNDERSTANDING OF ELIGIBILITY REQUIRMENTS Having received eligibility requirements from FEMA, what would you suggest to help them communicate with you more effectively?
|
If there is no other information or customer service suggestions, I would like you to change topics again and would like you to think about the project worksheet process that was performed. |
10 MIN
|
PROJECT WORKSHEET PROCESS:
7. PROJECT WORKSHEETS How was the FEMA field staff involved in the creation of project worksheets?
7a. MITIGATION FUNDING If you received Public Assistance mitigation funding, how satisfied were you with the process?
7b. IMPORTANT ASPECTS OF SITE VISIT(S) From your perspective, what aspects of the site visit were the most important to you?
7c. FEMA DEFINED SCOPE OF WORK AND DAMAGE REPAIR ESTIMATES If FEMA developed the scope of your projects or identified damage repair cost estimates, were you satisfied with the outcome?
7d. SUGGESTIONS TO IMPROVE PROJECT WORKSHEET PROCESS: Based on your experience, what do you think FEMA could do to improve the process involved in the project worksheet?
|
The project worksheet process leads us to another area of importance for most organizations recovering from a disaster. Now I would like you to think specifically about the actual financial assistance you received from FEMA. If you did not receive a monetary award from FEMA, we can discuss that as well. |
|
15 min |
PROGRAM RESULTS:
8. FEDERAL GOVERNMENT’S ROLE First, what should the federal government’s role be in assisting organizations in recovering from a disaster?
8a. SATISFACTION WITH FEMA’S FINANCIAL ASSISTANCE How satisfied are you with the level of financial assistance your organization received from FEMA in helping you recover from the disaster?
8b. FACTORS INFLUENCING SATISFACTION WITH ELIGIBILTY What factor’s do you consider when you determine your overall level of satisfaction with the assistance your organization received from FEMA?
8c. PROVIDING FUNDS Based on your recent experience in applying for public assistance, what do you feel constitutes fair and reasonable assistance?
|
For our last topic today, I would like you to think about the whole disaster recovery process. That is from the time of your Kickoff meeting until now.
|
5 min
|
9. SUGGESTIONS TO MEET NEEDS:
If you could give FEMA suggestions on how better to meet your organization’s disaster-related needs what would it be?
|
CLOSING: We are just about through with this session and before I give you this last exercise to complete, on behalf of FEMA I would like to thank you for your participation and assure you that the feedback you provided today will be used to improve service to disaster assistance victims across the country.
|
|
10. RANKING EXERCISE: Before leaving, on the graphic we have the top attributes you identified earlier as being the most important in doing business with FEMA. Before leaving, I would like each of you to take a few minutes to think about the attributes you indicated were the most important in interacting with FEMA. On the ranking sheet I have passed out, I would like you to rank the top seven of these attributes in their order of importance. When you have finished ranking these attributes, give me your ranking sheet as you leave. If you have any questions, please let me know.
|
ATTRIBUTE RANKING SHEET
|
|
|
MOST IMPORTANT |
1. |
|
|
2. |
|
|
3. |
|
|
4. |
|
|
5. |
|
|
6. |
|
7th MOST IMPORTANT |
7. |
|
COMMENTS:__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
INTERVIEW MODERATOR GUIDE: PUBLIC ASSISTANCE
SRIA Interview Guide
CUSTOMER SURVEY AND ANALYSIS SECTION
(CSA)
Pages 2-4 emailed to interviewees ahead of time to let them know purpose of interview, assure confidentiality, and what to expect. Information is again summarized at beginning of interview, and respondents are asked if they have any questions before interview begins.
Introduction
The Sandy Recovery Improvement Act, passed in January 2013, added Section 428 to the Stafford Act and authorized the Public Assistance program to implement alternative procedures pilot programs that affect the way that the PA program provides assistance for the repair, restoration, and replacement of damaged facilities (i.e., permanent work projects). Additionally, the PA Alternative Procedures set new guidelines to expedite and increase funding for debris removal projects. For permanent work, the law authorized FEMA to pilot the following alternative procedures:
Grants on the basis of fixed estimates for public assistance permanent work projects;
Alternate projects, also called in-lieu contributions, without the 10 percent reduction in eligible costs that is normally applied;
Consolidating the repair, restoration or replacement work on damaged facilities as a single project based on estimates; and
The use of all or part of the excess grant funds for cost-effective activities related to mitigation or PA program planning.
The law also emphasized that FEMA would accept cost estimates for fixed costs projects that were prepared by professionally licensed engineers; or at the request of a subrecipient, provide estimates to an independent expert panel for validation.
For debris removal, the law authorized FEMA to pilot the following alternative procedures:
Use of a sliding scale to determine the Federal share for removal of debris and wreckage, based on the time it takes to complete debris and wreckage removal;
Use of program income from recycled debris without offset to the grant amount;
Reimbursement of base and overtime wages for PA applicants performing or administering debris and wreckage removal; and
Cost-share incentive to a state, tribal or local government to have a debris management plan approved by FEMA and have pre-qualified one or more debris and wreckage removal contractors before the date of declaration of the major disaster.
Purpose of the Research
In order to understand the effectiveness of the Alternative Procedures program (also known as the Sandy Recovery Act, SRIA, Debris Pilot, 428 Projects, or PA Pilot), we are collecting insight from participants. The goals of the Alternative Procedures program are to increase flexibility for applicants, expedite the public assistance process, and provide financial incentives for timely and cost-effective completion of projects. We are interested in discussing your experience and feelings associated with participating in the Alternative Procedures program so we can identify successes and shortcomings in the new process. We are in no way evaluating or auditing your application for assistance. This is simply to better inform FEMA whether the changes made to the Public Assistance program are benefitting applicants.
Method
This research will involve your participation in a one-on-one interview that will take about one hour to complete. The Customer Satisfaction Analysis (CSA) Section created the survey design and carried out the study in order to be an independent, third party to conduct interviews and analyze data. Grounded Theory method will be utilized to conduct in-depth interviews and analyze data. Two members of CSA will be present during interviews, one will serve as interviewer, the other as a note-taker. Interviews will not be recorded.
Participant Selection
You have been selected to participate in this research because you took part in the SRIA procedures and can offer insight into the process. Participants have been taken from a sample all over the country to represent varying disaster types, locations, and regions. Information gained from participants will be compiled into reports, presentations, etc.
Voluntary Participation
Your participation in this research is entirely voluntary. It is your choice whether to participate or not. If you choose not to participate, there will be no bearing on current or future FEMA applications.
Procedures
A. We are asking for your help in learning about your experience in the Alternative Procedure process – how you made the decision to participate, comparing this process to previous disasters (if you have prior experience), and how the overall experience has been. Overall, we are looking for your opinion on the process, lessons learned, and future improvements for the program.
B. During the interview you will be asked a series of questions pertaining to your experience. These are opened ended questions which we will document your opinion. Though many people may have been involved in the process, we ask that you limit the interview to those most knowledgeable. We will not be recording the interview, and will need to take detailed notes during the interviews, too many people in the interview will make this very difficult.
If you do not wish to answer any of the questions during the interview, you may say so and we will move on to the next question. The information is confidential and only CSA will have access to the information documented during your interview.
Duration
We expect this interview to take 45 minutes to one hour.
Risks & Benefits
We don’t anticipate any risks from participating in this study. We hope to collect information to improve the PA program, and SIRA in particular, to enhance the program for future disasters.
Confidentiality
We will not be sharing information about you to anyone outside of the research team. The information that we collect from this research project will be kept private. Any information about you will be assigned a number instead of your name. Only the researchers will know what your number is. Your personal responses will not be attributed to you when we make our final reports for our FEMA counterparts, or outside of FEMA.
Sharing the Results
Results will be used for various reports within FEMA and to report to Congressional committees concerning the SRIA program.
Right to Refuse or Withdraw
You do not have to take part in this research and you can stop or withdraw at any time during this interview. Your answers will not affect the outcome of FEMA assistance, or future FEMA assistance.
Who to Contact
If you have any questions about this study, you can contact one of us or Chad Faber at (940) 891-8956 or at [email protected]
You can ask me any more questions about any part of the research study, if you wish to. Do you have any questions?
Notes:
For all – distinguish between debris removal and permanent work as necessary. They may call alternative procedures by a few different names: Alternative procedures, Sandy Recovery Act, SRIA, Debris Pilot, 428 Projects, or PA Pilot.
Capture transcript/verbatim, don’t worry about if it fits in the question that was asked – we can pull that out during analysis.
If a respondent does not want to answer/does not know a question, move on to the next one. The next question may jog their memory.
If the respondent has a question about their case, or a problem, ask if you can share their information and concern/problem (not their interview data) with someone who can help. We will pass their information along to the right person.
Assign an interview number based on state and sequential number for each state, ex. NY01.
Disaster #:
Interviewee Number:
Organization Represented:
Interviewer & Note taker:
Date and Time:
Location of Interview (City, State):
Which specific PA Alternative Procedures did the applicant use? Check all that apply:
|
Debris Removal |
|
|
Permanent Work |
||
|
|
Increased Federal Cost Share |
|
|
|
Subgrant Based on Fixed Estimate |
|
|
Recycling Revenue |
|
|
|
Consolidation of Fixed Subgrants |
|
|
Straight-time Force Account Labor |
|
|
|
Alternate Project (No Reduction in Costs) |
|
|
Debris Management Plan Incentive (2%) |
|
|
|
Excess Funds |
1. Please walk me through the process you went through to participate in the PA alternative procedures pilot program.
Probe: Who offered you Alternative (SIRA) Procedures? AND Why did you decide to participate in the program? If you did not participate, why not?
2 a. For those with previous disaster experience or some projects that use PA alternative procedures and others that don’t:
Comparing this experience to previous disasters, what is different from prior experiences? What is the same?
Probe: What challenges did you face pertaining to PA alternative procedures?
OR
2 b. For those without other disaster experience or projects that don’t use PA alterative procedures: What aspects of the PA alternative procedures did you find most valuable?
Probe: What do you see as the challenges to alternative procedures?
3. Would you recommend these procedures to others? Why or why not?
4. Thinking about future disasters, what could be done to improve or enhance the program?
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Brandi Lea |
| File Modified | 0000-00-00 |
| File Created | 2023-10-20 |