FEMA Manual FM-104-FY-22-101

CSA QUALITATIVE PROTOCOL

Individual Assistance (IA) Surveys

Disaster survivors’ needs and expectations evolve over time; therefore, the importance of conducting periodic qualitative research is essential to accurately measuring customer satisfaction. Focus groups and one-on-one interviews assists in many areas such as the re-design of the survey questionnaires, provide FEMA with an understanding of how to improve information and correspondence with registrants for disaster assistance, how to improve programs and processes for disaster survivors, as well as how to improve the in-person, on-the-phone and on-line experiences.  

This protocol was developed to provide consistent guidelines within the Customer Survey and Analysis (CSA) Section of FEMA who plan to conduct qualitative studies for issues related to the Individual Assistance (IA) surveys which include: Initial, Contact, and Assessment.

There are three phases to conducting a focus group and these phases are explained below:

PHASE 1: BEFORE THE QUALITATIVE STUDY

1. Meet with management/leadership to identify overall purpose and outcomes for the focus group/interview.

   • Discuss possible hypotheses.

   • Identify funding sources.

2. Determine target population and number of responses needed.

• Multiple focus groups will allow for comparisons and to examine themes that emerge from each discussion.

• Past results show focus groups average 6% response rate from the initial call to participate. For each of the 4 programs, there is an annual allotment of 960 unique participants. This includes 12 participants per 5 days of separate sessions per 4 geographical location.

• Focus groups are approximately 2 hours each (plus 1 hour travel).

3. Identify participants:

• Determine how many participants are needed in each focus group. Each focus group should have between six and twelve participants. Fewer than six participants may limit the conversation and yield poor data while more than twelve can be unmanageable. We allow for maximum of 12 per session with the anticipation that some may not attend.

• Develop a list of key attributes to seek in participants based on identified purpose.

• Secure names and contact information and send invitations.

4. Generate questions.

• Based on the purpose and goals of the focus group, identify no more than 5-6 questions.

• Revisit the questions to make sure they will promote appropriate discussion to address research questions.

• Order the questions from general to specific.

• Scope of questions from each stage in the registration to determination process may include topics related to the following survey:

• Initial

• Recent changes to registration process.

• Technology/ Usability with the online registration tool.

• Application process and resources.

• Disaster recovery center (DRC) changes/usability/expectations.

• Contact

• Recent changes related to correspondences and information.

• Inspection process: Transparency and understanding.

• Expectations of inspections.

• Technology/ Usability with the online/automated process.

• Assessment

• Repair quality and cost.

• Communication and usability.

• Assistance expectations.

• Recovery process.

6. Select a facilitator and note-taker

• The facilitator should not be someone who directly oversees the issue being researched. This may make participants less open to sharing their thoughts or concerns.

• The facilitator should be able to keep the discussion going, deal tactfully with difficult or outspoken group members, and make sure all participants are heard.

• The facilitator should ask the questions and probes, but not participate in the dialogue or correct participants.

• If possible, a program analyst can sit in for specific program questions.

7. Choose the location

• Choose a location which is comfortable, easily accessible, and where participants can see one another.

• Choose a setting which does not bias the information gathered. Preferably libraries or other public forums with meeting rooms.

8. Develop script: Moderator’s Guide (See Appendix A)

• Part one: Welcome participants and make introductions. Provide logistics of building related to emergency exits and restrooms and sign consent forms.

• Part two: More about FEMA CSA and reasons for the focus groups. Explain that information is confidential and no names will be used. Provide FEMA Helpline number.

• Part three: Ask for introductions from the group.

• Part four: Ask prepared questions; remember to use probes and follow up questions to explore the key concepts more deeply.

• Part five: Close the focus group – thank participants, give them contact information for further follow up if requested, and explain how data will be analyzed and shared.

PHASE 2: CONDUCT THE QUALITATIVE STUDY

1. Bring materials:

• Notebook/computer

• Flip chart paper/ no board is available (if available)

• Focus group list of participants

• Focus group script

• Name tags

• Location signage

• Watch or clock

• Consent forms

2. Arrive before the participants to set up room, display signs directing participants to room, etc.

3. Introduce facilitator and the note-taker (if applicable) and carry on the focus group

according to the script (See Appendix A).

4. Conduct the session, being mindful of the following:

• Set a positive tone while empathizing with their situation as a recent disaster survivor.

• Make sure everyone is heard; draw out quieter group members.

• Probe for more complete answers.

• Monitor questions and the time closely – it is the job of the facilitator job to make sure the group is on track.

• Don’t argue a point with a participant, even if they are wrong. Address it later if needed.

• Thank participants and let them know FEMA values their input considering the disruption in their lives when dealing with a disaster.

• (See Appendix A)

PHASE 3: INTERPRETING AND REPORTING THE RESULTS

1. Summarize each meeting

• Immediately after the meeting, the facilitator should write up a quick summary of his/her impressions.

• Transcribe the notes of the focus group. This should be done as soon as possible after the focus group has been conducted.

• If multiple facilitators have been used for two or more groups, discuss impressions with the other facilitators before reviewing the transcript.

2. Analyze the summaries

• Read the notes and look for themes/trends. Write down any themes which occur more than once.

• Context and tone are just as important as words. If comments are phrased negatively or triggered an emotional response, this should be noted in the analysis.

• Interpret the results

  • What are the major findings?

  • What would be possible recommendations?

3. Write the report

• The report should include the research purpose, outcomes, process, findings (major themes and minor themes), and recommendations.

• Submit the report to management.

4. Make adjustments/take action on what was learned (After-action review)

• Schedule a meeting with management and/or study facilitators to discuss the implications.

  • Highlight the main themes, and/or issues that arose in the study.

    • These themes/issues can be related to logistics of focus group/interviews or program discussion.

  • Prioritize the results and make action plans for the most important priorities.

Appendix A

Moderator’s Guide Example

IHP LETTERS REVISION

FOCUS GROUP

MODERATOR’S GUIDE

Applicant Feedback for the Revision of the IHP Letters

CUSTOMER SURVEY and ANALYSIS (CSA) SECTION

Example

IHP LETTERS REVISION FOCUS GROUP

MODERATOR’S GUIDE

4 min About Us Reason Note-taking Personal Opinion Ease and Assurance No case or program info FEMA #		PURPOSE / SESSION DETAILS Session Details: About us Reason we Invited Participants Note Taking Personal Opinions/Ease/Assurance Ease and Assurance No Case Info FEMA Helpline number Now a little bit about us. We work in various roles in the Recovery Division of FEMA, providing disaster assistance to individuals like you and your families who have been in a disaster. Blair is the Section Chief of the Program Management Section. Anna is an analyst who helps make improvements to the services FEMA provides. And Sheri and I are analysts who collect disaster survivors’ opinions about the services FEMA provides. The reason we invited you here today is to ask you to review the information letters FEMA has redesigned and then ask you to answer a few questions about the new design. Specifically, we want you to tell us if they are easy to read and easy to understand. Sheri will be typing your statements as you speak so that we have an accurate record of your responses. With that in mind, we ask that you speak clearly and one at a time so she doesn’t miss anything, and so everyone has an opportunity to speak. There are no right or wrong answers. This discussion is meant to gather your personal opinions which will help us determine any areas in the letters that may need more attention. We want you to feel completely at ease to share your true thoughts, so please be assured that your responses today will not in any way affect the outcome of your FEMA case. I also want to mention that we do not have any information about the decisions of your case. Our purpose here today is simply to gather your opinions about the disaster assistance information that is provided in the new letters. If you have questions about your case, we encourage you to call the FEMA Helpline at 1-800-621- FEMA (3362).
	3-6 min (30 seconds per participant) 3 min 3 Letters Original Packet New Intro Letter Super Letter		PARTICIPANT INTRODUCTIONS Okay, now I’d like each of you to introduce yourself to the group. LETTER DETAILS 3 Letters New 7 page Introduction Letter We will be looking at 3 letters: a new introduction letter, and 2 new approval status letters. The first documents you received after you applied with FEMA were in a packet containing a 2 page introduction letter, a 9 page list of referrals to other agencies, and a 33 page booklet titled the Applicant’s Guide to the Individuals & Households Program. [hold up copies of each to show] The information in these 44 pages has been shortened to one 5 page introduction letter and one 2 page referral document. [hold up copies of each to show] I will ask you to review the 5 page letter and then ask you to answer some questions about it. Next is the approval letters. After you applied you received a letter explaining what assistance you were approved to receive and/or not approved to receive, and how to appeal if you disagreed with the decisions. You also may have received additional letters explaining the results of any appeals you may have requested. We have redesigned these letters and I will ask you to review them and then again ask you to answer some questions. New Super Letter and A-Super Letter

10 min Reading Preferences

QUESTIONS - Top of Mind Opening Question Before we begin with the letter reviews, I will start with a general question. For this first question I will go around the room to give each of you an opportunity to share your opinions one at a time. 1. Which is more important to you when reading information about any type of service being provided to you: do you prefer brief reading materials that you can read quickly in one sitting, or detailed materials that might take you longer to read but fully answer all of your questions? ________, let’s start with you…

10 min Read in Sections Wanted to Know Page 1 Two Parts Easy to Read and Clearly Explain

QUESTIONS – Introduction Letter We are ready to review the new introduction to FEMA letter now. You will read the letter in sections and I will ask a question after each section. The letter is printed front and back which is the way it will be printed when it is mailed out. While reading, I would like you to think about what you most wanted to know about disaster assistance after you registered with FEMA. Read all of page 1, then flip it over and read until you get to the purple flag on page 2. [allow 5 minutes to read] 2. Tell us if the descriptions of the categories in the Individuals and Households Program were easy to read, and if they clearly explained how FEMA’s disaster assistance works. Who would like to go first?

10 min Insurance Inspection SBA Appeal

Thank you everyone. Now you can read the rest of page 2 and continue to page 3, stopping at the green flag. [allow 5 minutes to read] 3. After reading the details in this section, what is your understanding of what happens after applying for disaster assistance? Who would like to go first?

10 min Flood Insurance

Next read the section at the bottom of page 3 under the green flag and stop at the end of the page. [allow 3 minutes to read] 4. What do you understand about Flood Insurance after reading this section? Who would like to start?

10 min Mail Record Keep Fraud Discrimination

Now read all of page 4 and stop at the end of the page. [allow 5 minutes to read] 5. What do you think about the way the topics on this page are explained? Who would like to start?

10 min Resources

Now read all of page 5 and stop at the end of the page. [allow 5 minutes to read] 6. What do you feel about the level of detail on this page? Who would like to start?

Closing That was the last question! You have all provided excellent feedback. We thank you for your participation and assure you that the feedback you provided today will be used to improve service to disaster assistance victims across the country. [collect hard copies]

10