Appendix O Pretest Methods and Summary of Findings_01202022

Appendix O. Pretest Methods and Summary of Findings

Beginning July 19, Insight pretested the following instruments for the study of food security and well-being in Puerto Rico:

• Household survey

• In-depth interview protocol

• Recruitment materials

The overarching objective of the pretest was to ensure the instruments were clear and understandable to respondents. Specific pretest objectives for both instruments follow:

• Identify problems related to communicating intent or meaning of questions.

• Determine whether respondents can accurately provide the information requested.

• Evaluate the flow, question order, and respondent burden in terms of the number of questions and recall of difficult events such as natural disasters.

• Ensure the Spanish translation conveys the intended meaning.

The survey pretest also aimed to—

• Assess the response options for relevance and adequate range of response options.

• Identify problems with introductions, instructions, or explanations.

• Assess the cultural relevance of the questions.

1. Recruitment Methods

To identify potential pretest participants, we developed a recruitment announcement for distribution via social media by Estudios Técnicos and the Food Bank of Puerto Rico. The announcement included a point of contact and telephone number for a team member from Estudios Técnicos who screened callers for eligibility and scheduled the interviews.

We planned to complete nine pretest interviews for each data collection instrument, with a ratio of five NAP participants to four low-income nonparticipants. Within those two subgroups, we tried to recruit a mix of households with children and without children. Finally, for each instrument, we tried to identify two participants who could complete the pretest in English.

We developed a recruitment script with questions about age, household composition, income, and NAP participation status to determine each person’s subgroup status. We also asked about language preference to identify individuals who could complete the pretest in English.

Forty-seven individuals responded to the recruitment announcement. Of those, 39 were eligible and 8 were ineligible based on income or age. The recruiter scheduled appointments with eligible individuals, tracking each person’s NAP participation status and household composition to ensure all subgroups were represented. She also made reminder calls or sent reminder text messages to each pretest participant the day before the scheduled interview to confirm their appointment. Table 1 below shows the distribution of volunteers and completed interviews by subgroup.

Table 1. Number of Volunteers Screened and Interviews Completed by Subgroup

Group Status	Screened	Completed
NAP with children	15	7
NAP without children	14	7
Non-NAP with children	4	1
Non-NAP without children	6	2
Ineligible	8	0
Total screened	47	17

Tables 2 and 3 show the distribution of pretest respondents by subgroup for the survey and in-depth interview instruments, respectively. Seven of the eight survey pretest interviews were conducted in person at Estudios Técnicos. The eighth pretest interview was conducted in person at the home of the pretest participant. This participant lived on the island’s west coast, where one of the pretest interviewers planned to be after completing the interviews in San Juan.

We originally planned to conduct all in-depth interviews in person. We started offering participants the option of participating via Zoom after several scheduled participants canceled their in-person appointment. Several factors contributed to these cancellations. A truckers’ strike resulted in long lines at gas stations and increased traffic as residents rushed to fill their gas tanks and stock up on necessities. A few pretest participants did not want to risk sitting in traffic or running out of gas while traveling to Estudios Técnicos. Other participants had a change in their work schedule or could not find childcare. Of the nine completed in-depth interview pretest interviews, four were conducted in person and five via Zoom.

It is worth noting that only four volunteers were non-NAP participants with children. We were not able to complete a survey pretest interview with anyone from this group, and the one in-depth interview we completed was via Zoom. There are several possible reasons this subgroup might have been particularly difficult to recruit. Widespread loss of work and income because of the pandemic has increased participation in NAP, thereby reducing the number of nonparticipating households that met our low-income threshold. Without NAP support, adults in these households may be working multiple jobs to support their families, leaving them with limited time to participate in the pretest. If economic circumstances at the time of data collection are similar to current circumstances, these findings point to possible challenges that could affect response rates among this critical subgroup. The study team will closely monitor response rates by subgroup during the field period and is currently considering strategies for bolstering participation rates among subgroups that are slow to respond.

Though we identified two volunteers who were able to complete the pretest in English (one survey respondent and one in-depth interview respondent), no one expressed a preference for English. Based on this finding, we expect most respondents will complete the survey in Spanish and request that the interview be conducted in Spanish.

Table 2. Number of Respondents per Subgroup: Survey Pretest

NAP Participation Status	Interview Language		Total
	Spanish	English
NAP with children	3	0	3
NAP without children	3	1	4
Non-NAP with children	0	0	0
Non-NAP without children	1	0	1
Total	7	1	8

Table 3. Number of Respondents per Subgroup: In-Depth Interview Pretest

NAP Participation Status	Interview Language		Total
	Spanish	English
NAP with children	4	0	4
NAP without children	3	0	3
Non-NAP with children	1	0	1
Non-NAP without children	0	1	1
Total	8	1	9

2. Pretest Interviews for Survey Instrument

Two bilingual interviewers from Insight conducted eight pretest interviews using the “think-aloud” method. This method involves asking the participant to read each question aloud and verbalize their thought process as they read the response options and select their answer.

At the start of the pretest interview, the interviewer explained the purpose of the study and the aims of the pretest. She also reviewed the consent form, which stated that participants’ individual responses would not be shared with anyone outside the research team, but they might be reported in a memo in combination with other participants’ responses and without any personal identifiers to explain any recommendations for improving the data collection instruments. The consent form also described the respondents’ rights as research subjects and the voluntary nature of their participation, emphasizing that their decision to participate or not participate would not affect their NAP benefits. Finally, the interviewer asked the respondents’ permission to audiorecord the interview and asked for verbal consent to proceed with the interview. All respondents agreed to have the interview recorded.

Interviewers introduced the think-aloud method by engaging respondents in a practice exercise. To help the respondent get in the habit of narrating their thoughts as they formulate their answer to a question, the interviewer asked the participant to identify the number of windows in the place where they live. The interviewer encouraged the participant to envision the space, counting each window as they moved from one wall to another. Once participants finished this exercise, the interviewer asked them to begin the survey.

1. Survey Length

The duration of the survey pretest using the think-aloud method ranged between 80 and 115 minutes. No participants expressed complaints about the length of the survey. These volunteers were informed ahead of time that the pretest interview would take up to 90 minutes, including time to debrief and discuss specific survey items. One of the pretest interviewers identified two NAP participating volunteers who agreed to complete the survey without interruption. One volunteer completed the survey in 38 minutes. The second volunteer completed it in 45 minutes, for an average time of 41 minutes. It is possible randomly chosen households will be less willing to complete a survey of this length. For this reason, we have looked for opportunities to streamline the questionnaire.

2. General Issues/Impressions

Participants had little trouble completing the survey, and they had thoughtful suggestions regarding the response options. For example, when answering A.8., several participants suggested the list of food categories did not reflect the typical Puerto Rican diet. There were a few places where one or more respondents overlooked the instructions or a skip pattern. We will address this issue when designing the final print layout of the paper survey, but for now, we have added instructions in bold font, all caps.

Pretest participants did not refuse to answer any questions, and they did not consider any items to be especially sensitive. Two pretest respondents brought a child with them to the pretest interview. The interviewer skipped the food security module for those two participants based on concerns that the participants would not feel comfortable reading and answering aloud questions about skipping meals in front of their children. This concern should not be an issue during the data collection because sample members will have the option of responding to the survey privately via the web, on paper, or by phone.

3. Item-Level Findings and Recommendations

The table below lists questionnaire items, including those for which respondents reported difficulty or confusion or provided additional comments. The questions appear in numerical order for ease of reference. The table describes the nature of the problem and resulting changes.

Item Number (Pre/post-test item number if re-numbering occurs)	Question Stem	Findings/Observations	Resulting Changes
Section A
A.1.	Are you the primary shopper for your household (the person who usually buys most of the groceries)?	No issues	No changes
A.2.	Where do you (or the primary shopper) usually buy most of your groceries? MARK ONE	Participants were not sure if the question was asking where they shop most frequently or where they buy most of their food. For example, they might pick up a few things several times a week from a local store but visit a warehouse-type store once a month to purchase food in bulk.	Be specific about selecting one option. Delete “usually” to make it clear we are asking where the primary shopper buys most groceries.
A.3.	In a typical month, how often do you (or the primary shopper) shop for food at this store?	No issues	No changes
A.4.	How much time does it usually take you to get to the store?	No issues	No changes
A.5.	About how many miles do you (or the primary shopper) live from the store where you buy most of your groceries?	Participants estimated the distance based on how much time it takes them to get to the store.	Delete the question because respondents are coming up with an answer based on the time it takes to get to the store. Estimated distance may not be accurate.
A.6./A.5.	How do you (or the primary shopper) usually get to this store?	A few other methods were mentioned, including scooters and skateboards, but these would fall under “Some other way.”	No changes
A.7./A.8.	How often do you (or the primary shopper) usually buy groceries at any other stores?	No issues	No changes
A.8./A.6.	Thinking about the stores where you (or the primary shopper) shop for groceries, please indicate the extent to which a variety (that is, different kinds) of products in these food categories are available to you at these stores.	Participants had to read this question several times because the question stem is long and they found the wording confusing. Asking about availability of a variety of foods in multiple categories may be too cognitively complex. One participant reported frequency with which he bought food in each category instead of the variety. The examples in some of the food categories are not commonly consumed in Puerto Rico. Food categories should be revised.	Change question to: A.6. How would you describe the store’s selection of foods in the categories listed below? For each category, please indicate if the store’s selection is poor, average, or good. If the store does not sell the food item, please select not applicable. a. Fresh fruits b. Frozen fruits c. Canned fruits d. Fresh vegetables e. Frozen vegetables f. Canned vegetables g. Dairy products such as milk, cheese, or yogurt h. Meats, such as beef, chicken, or pork i. Seafood
A.9./A.7.	Thinking about the stores where you (or the primary shopper) shop for groceries, how easy is it to afford these foods on your budget?	Literal translation of “afford” is not a word commonly used in Puerto Rico; it is clearer to say “buy on your budget.” Food categories should be revised per A.8.	Change “afford” to “buy.”
A.10./A.9.	Is shopping for groceries for your household difficult for any of the following reasons? SELECT ALL THAT APPLY	Weather can be another barrier for individuals who do not have a car; parking can also be a barrier. Respondents wondered why “None of the above, shopping for groceries for my household is not difficult” was set apart from other response options. The intention was to have the respondent select none of the above OR mark all reasons that apply.	Drop “none of the above” and instruct respondent to respond “yes” or “no” for each item.
A.11./A.10	In the past 30 days, about how much money did you/your household spend on food at supermarkets, grocery stores, or other stores that sell food products (including any purchase made with PAN)?	Pretest participants provided a dollar amount, but not everyone included what they spend using PAN benefits. A few participants also asked why the survey does not ask about the last month instead of the last 30 days.	Add instructions in bold caps to indicate what respondents spend on food with their NAP benefits. Though “last month” is shorter than asking “in the last 30 days,” the majority of questions that include a reference period specify number of days (e.g., “the last 30 days” or “the last 12 months”). The food security module in particular references the number of days, and that instrument is well validated. To be consistent, we will continue to refer to the number of days.
Section B
		Respondents experiencing very low food security may have concerns reporting any difficulties feeding their children and will want to know their answers will not be reported to any agencies.	Add a statement indicating responses will be kept private and will not have any impact on their benefits.
Section C
C.1.	In the last 12 months, how often did you or people in your household have to do any of the following things to make your food money go further?	(e) Direct translation of “exchange labor for food” was not easily understood by participants. (g) Get food from a pantry or soup kitchen: Food pantry or soup kitchen are not used in Spanish. (h) Skip buying medicine or seeking medical attention: Some participants indicated they might do this for themselves or another adult in the household, but they would not skip medical care for their kids.	(e) Instead of “exchange labor for food,” Spanish translation will be closer to “accept food as payment for work.” (g) For “Food bank or pantry,” use “banco de alimentos o alacena/comedor comunitario.”
C.2.	How confident are you that your household could come up with $400 if an unexpected expense arose within the next month?	No one understood this question. Participants were only able to comprehend it after the interviewer reframed the question, asking: if you had an emergency and you needed $400, would you have that money, or could you somehow get it—for example, by borrowing it?	This question comes from the Federal Reserve’s Survey of Household Economics and Decision Making (and has been included in other surveys, such as the Urban Institute’s Well Being and Basic Needs survey) to assess financial fragility. Because the survey includes several other items that measure financial fragility, we suggest dropping this item.
C.3/C.2.	Does your neighborhood have any of the following places or programs for households that may need help with food or meals? Food bank or pantry Free meals served at a shelter, food kitchen, or soup kitchen Free meals served at a church, community, or senior center Meals on Wheels or other home delivery meal programs	Some respondents said they have not been to any of these places recently, so they wanted to see a “don’t know” option. When asked if they would add any other places or programs to the list, several participants suggested the food box program. This might be what FNS calls the Commodity Supplemental Food Program. Most residents will not be familiar with that program name, though, and they may not know who operates the program, so we recommend using commodities or food box program.	For “Food bank or pantry,” use “banco de alimentos o alacena/comedor comunitario” per comment on C.1.g. Add “Don’t know” as a response option. Add “Food boxes” as a response option.
C.4./C.3.	In the last 12 months, have you or anyone in your household visited a food bank, pantry, or similar place to get food for you or your household?	Make sure the translation for food pantry is consistent throughout.	Change references to food banks or pantries to “banco de alimentos o alacena/comedor comunitario” per comment on C.1.g.
C.4a./C.3a.	Please indicate how much you agree or disagree with the following statements about the neighborhood where you live and the people around you.
C.5./C.4.	People around here are willing to help their neighbors.	No issues	No changes
C.6./C.5.	This is a close-knit or “tight” neighborhood where people generally know one another.	No issues	No changes
C7./C.6.	If I had to borrow $30 in an emergency, I could borrow it from a neighbor.	No issues	No changes
C.8./C.7.	People in this neighborhood generally don’t get along with one another.	No issues	No changes
C.9./C.8.	People in this neighborhood can be trusted.	No issues	No changes
C.10./C.9.	If I were sick, I could count on my neighbors to shop for groceries for me.	No issues	No changes
C.11./C.10.	People in this neighborhood do not share the same values.	One person said she had to read it a few times because she was not used to negative statements.	This item is part of a validated scale, and it might have been structured this way to help prompt people to stop and think about their answer. We recommend keeping the item in its standard form.
The next questions are about strategies some households may use to meet their basic needs after a natural disaster. Natural disasters may include hurricanes, tropical storms, drought, wildfires, and earthquakes.
C.12./C.11.	Have you experienced a natural disaster in Puerto Rico in the last 5 years?	One person checked all three disasters.	Add instructions in bold all caps to select one option. Specific response options will be updated when the survey goes to print to reflect more recent disasters.
C.12a./C.11a.	Please indicate which disaster you most recently experienced.	C12a: One person suggested to add “flooding.”
C.13./C.12.	Thinking about the most recent natural disaster your household experienced, did your household do any of the following to obtain needed resources?	Several participants asked why FEMA is not listed. Others suggested we ask about applying for NAP.	Change item C.12.a to “Apply for or rely on support from a Government agency (e.g., FEMA, ADSEF) or nonprofits (e.g., Red Cross).”
Section D
D.1.	Including you, how many people currently living in your household are— [ __ ] Younger than 18 [ __ ] Between 18 and 59 [ __ ] Aged 60 years and older	Definition of household works. Participants were easily able to count household members. One participant only reported her kids.	Put “including you” in bold caps.
D.2.	In the last 12 months, has there been a change to the number of people living in your household?	Everyone said no, possibly because of pandemic; no one is moving around.	No changes
D.2a.	What caused this change?	No issues	No changes
D.3.	Which best describes the place where you live?	No issues	No changes
D.4.	Is the place where you live— [ ] Owned by you or someone in this household with a mortgage or loan? [ ] Owned by you or someone in this household free and clear (without a mortgage or loan)? [ ] Rented? [ ] Occupied without payment of rent?	No issues	No changes
D.5.	Are there any other households living at your same mailing address? This might include households living in garage or basement apartments.	No issues	No changes
D.6.	Do you or any member of your household have access to the internet using a—	Some participants marked only one item instead of selecting “no” for options they do not have.	Add instructions: “For each item below (a–e), mark yes or no” in all caps.
D.7.	What language is most commonly spoken in your home?	No issues	No changes
D.8.	Thinking about the person in your household who has completed the most schooling, what is the highest degree or level of school that person completed?	A few participants wondered why there were so many categories. One suggestion was to separate by school transitions (elementary to middle school, middle school to high school, and high school to college/HS) because students tend to drop out at these points.	Recommend reverting to prior list of categories, which is less detailed and takes up less space on the questionnaire. We considered the transitions but concluded we would likely collapse less than high school into one category for analytic purposes.
Section E
The following questions ask about the participation of your household in various Federal programs.
E.1.	Have you or anyone in your household received benefits from Programa de Asistencia Nutricional (PAN) in the last 12 months? PAN benefits are provided on an electronic debit card called the Tarjeta de la Familia.	No issues	No changes
E.1.a	During the past 12 months, for how many months did you get PAN benefits?	No issues	No changes
E.1.b	How many weeks do your monthly PAN benefits usually last?	No issues	No changes
E.1.c	Do you or anyone in your household currently receive benefits from PAN?	No issues	No changes
E.1.d	How much did you receive in PAN benefits last month (or the last month you received benefits if you didn’t receive benefits last month)?	No issues	No changes
E.1.e	Did you or anyone in your household receive benefits from PAN in (insert month, year of sample selection)?	Respondents found it confusing because no date was included in parentheses. When the survey is fielded, we will include the date ADSEF produced the file.	No changes
E.2.	In the past 12 months, did you or anyone in your household receive—	One person did not know what LIHEAP was. One person said yes to WIC because her kids got it when they were young.	No changes
Section F
	The following questions ask about household employment and finances. These responses will only be reported in summary format and will not affect your receipt of any public benefits.
F.1.	Including yourself, how many adults aged 18 and older in the household were employed in the last month?	No issues	No changes
F.2./F.2a.	Has anyone in your household had a change in employment or a change in pay or hours worked at a job in the past 6 months? [ ] Yes [ ] No Go to F.3 Was that change your household experienced in the last 6 months because of— (Fill in all that apply) [ ] Getting a job [ ] Losing a job [ ] Increase in pay or hours [ ] Decrease in pay or hours	F2a. One participant had some trouble selecting an answer for himself because he is a seasonal worker. Another participant had difficulty selecting a reason; she had to leave her job because she was sick.	Change “losing a job” to “losing a job or leaving a job.” Simplify F.2.a to “Was that change because of—"
F.3.	Are there any 16- or 17-year-old youths in your household?	No issues	No changes
F.4.	Please indicate whether you or anyone in your household received income in the last year from any of the following: MARK ALL THAT APPLY	One person suggested commissions, bonuses, and tips should be in a separate category from wages and salary.	No changes; It is not clear how we would use the information we would derive from separating the first response option into two parts. This item is taken from the Puerto Rico Community Survey. The question wording and response options are the same, with one exception: the PRCS lists Social Security Insurance as one of the response options. We deleted this option because residents of PR do not receive SSI. We contacted the Census Bureau to ask why SSI is listed and received the following response: Puerto Ricans can legitimately record SSI if they were living in the states and receiving SSI during the past 12 months but moved back to Puerto Rico when answering the ACS. For the purposes of this study, we recommend keeping SSI off the list of income sources.
F.5.	Which category best describes your total household income last year before taxes or other deductions? MARK ONE	One person noted this question asks about a year; they asked why we do not ask the same elsewhere instead of using “12 months.”	No changes; “last year” implies last calendar year, and we are looking for an estimate of annual income.
F.6.	Which of the following best describes your household’s current financial condition? MARK ONE	No issues	No changes
F.7.	Please rate the extent to which each of the problems below was a concern for your household in the past year.	Paying for cellular phone or internet should be included.	Added “Ability to pay for cellular phone or internet service.”
Section G
G.1.	Please indicate if you or anyone in your household has difficulty with the following activities.	No issues	No changes
G.2.	Are you or is anyone in your household limited in the kind OR amount of work you can do because of a physical, mental, or emotional problem?	No issues	No changes
G.3.	Do you or does anyone in your household have difficulty doing errands alone, such as visiting a doctor’s office or shopping, because of a physical, mental, or emotional condition?	No issues	No changes
G.4.	Are any members of your household currently covered by any of the following types of health insurance or health coverage plans? a. Insurance through a current or former employer or union b. Medicare, for people 65 and older or people with certain disabilities c. Medicaid, Medical Assistance, Children’s Health Insurance Program (CHIP), or any kind of Puerto Rico-sponsored assistance plan based on income or a disability d. TRICARE or other military healthcare e. Any other type of health insurance or health coverage plan	No issues	No changes
G.5.	In the past year, did anyone in your household have problems paying any medical bills?	No issues	No changes
G.6.	Thinking about your healthcare experiences over the past year, has medical care been delayed for anyone in the household because of worry about the cost?	No issues	No changes
G.7.	In the past year, was there any time when anyone in the household needed medical care but did not get it because they couldn’t afford it?	No issues	No changes
G.8.	In the past year, have you or any members of your household received treatment, counseling, or services because of a problem with alcohol, tobacco, or drug use?	No issues	No changes
Section H
H.1.	How old are you?	No issues	No changes
H.2.	Do you currently describe yourself as male, female, or transgender?	One participant asked why the response options are included in the question.	Drop “male, female, or transgender” from the question stem to make it more straightforward.
H.3.	What is your marital status?	One person suggested “single” should be an option.	No change; the current response options are taken from the PRCS, and “single” is not a mutually exclusive category.
H.4.	Are you of Hispanic, Latino, or Spanish origin?	In the Spanish translation, “Spanish” raises confusion.	Ask about Hispanic or Latino.
H.5.	What is your race? (Select all that apply)	It was suggested to add “other.”	Added “other.”

3. Pretest Interviews for the In-Depth Interview Protocol

1. Interview Length

The in-depth interviews lasted between 58 and 100 minutes. With one exception, all pretest interviews took less than one hour. The longer interview was with a respondent who took a few brief breaks during the interview to collect her thoughts. We estimate the in-depth interviews will take no more than one hour.

1. General Issues or Impressions

Overall, results of the in-depth interviews indicated that the questions were clear, appropriate, and easily answered by participants. Based on feedback from interviewers and respondents, we recommend only minor revisions. Suggested revisions are summarized below.

Interviewers observed that the progression of questions helped ease participants into talking about potentially sensitive subjects, and they found participants to be forthcoming when asked about how they have coped with natural disasters and COVID-19. Some participants said no one had ever asked them about those experiences, and they found it worthwhile to reflect on what happened, how these experiences affected them, and how they coped. One participant was visibly upset when describing her experience of the earthquakes. The interviewer offered to take a break or stop the interview, but the participant wanted to continue. She was among those who thanked the interviewer for taking an interest in her story. These encounters have important implications for interviewer training, as described below.

Many participants considered their recent experiences of food security atypical, and they attributed their relative comfort with purchasing food to COVID-19 relief efforts. They were concerned, however, about how long those relief efforts might continue and how they would manage if NAP expanded benefits were discontinued.

3. Section-Level Findings and Recommendations

Section A. You and Your Household

Beginning the interview with questions about household composition worked well, as participants easily identified who lives with them. The questions provide an effective “ice breaker,” and the information gathered will help address research questions about hunger-coping strategies among intergenerational households.

Questions 1a and b elicited brief answers, but interviewers thought the questions were useful in getting participants warmed up to the interview process.

Section B. Community Food Needs

This brief section is intended to gain insight into the factors that contribute to low food security in a community (e.g., having money for food, getting to stores) and if particular households are more likely than others to experience low food security (e.g., single-parent households). Some responses reflected a desire to respect neighbors’ privacy; for example, several participants said they were not in other people’s business, so they did not know how to answer. Interviewers offered suggestions for rewording the questions to make it clear we are not asking anyone to identify specific people. As part of the interviewer training, we will discuss possible probes interviewers can use if a participant hesitates to answer. Rather than scripting these probes in advance, we will ask interviewers to offer suggestions in their own words and discuss their suggestions as a group. The purpose of the discussion will be to ensure all interviewers have a common understanding of what we hope to learn from the question and share ideas on how to encourage participants to talk openly about economic circumstances in their community.

Section C. Meeting Family Food Needs

Participants provided detailed information about their shopping habits, preferences, and strategies. In response to the question, “Are there any food stores or markets nearby that you don’t shop at?” a few people mentioned the Mercado Familiar. El Mercado Familiar (Family Market Program) is a collaboration between ADSEF and Puerto Rico’s Secretary of Agriculture intended to stimulate local food production and increase NAP participants’ consumption of fruits and vegetables. A few participants reported they no longer take advantage of the program because the hours are limited, the produce is expensive, and the portion of their NAP benefit they can use at these markets is not enough to purchase what they would need to feed their household.

In response to the question, “Have there been times recently when you did not have enough food or were worried about how to stretch your food dollars?” many participants said such worries were greatly reduced since they started receiving additional assistance as part of ongoing COVID-19 relief efforts. Participants said they frequently worried about having enough money for food prior to COVID-19, and they discussed strategies they used to stretch their food dollars, such as shopping for items in bulk or getting fruits or vegetables from neighbors with a garden.

Participants were uncertain how to answer the question, “What are the most important things that have helped you to get by during difficult times?” Some said they relied on their faith or felt motivated to be strong for their family, but they did not identify practical strategies that proved helpful. Participants found it easier to answer when the question followed a discussion about coping with disasters or COVID-19 (question 23). For this reason, we recommend deleting the question from section C.

Section D. Life Events and Coping With Challenges

Participants were very forthcoming with information when asked about their experience with natural disasters and COVID-19, and several said they appreciated the opportunity to talk about these topics. One participant was moved to tears as she recounted the traumas she experienced after the earthquakes. The interviewer asked the participant if she wished to take a break and offered to do some deep breathing exercises with her. With some brief breaks, the participant was able to complete the interview, and she thanked the interviewer for listening. She said she did not realize those experiences were still so fresh in her mind. When the study team debriefed after this interview, we discussed some implications for interviewer training. Specifically, we will train all interviewers on the use of trauma-informed interviewing skills and provide them with referral resources to ensure they are well prepared to handle similar situations.

Sections E and F. NAP

When asked how long their NAP benefits typically last, participants distinguished between pre- and post-COVID; as mentioned previously, participants said that pre-COVID it was difficult to make their benefits last a full month, but this situation started to change after the pandemic, when stimulus checks were issued and other support became more available.

When asked, “What are some reasons people might choose not to apply for NAP?” respondents were able to offer a few reasons, some based on stories they had heard about others’ experience with the application or eligibility determination process. For example, one participant said she has an elderly neighbor who was denied eligibility based on income, even though she spends a large proportion of her income on medicine. The participant noted that such stories discourage people from applying because they create a perception that the eligibility determination process is unfair, cryptic, or complicated.

4. Recruitment Materials

Pretest interviewers asked participants to read and comment on the survey invitation letter for the area probability survey. There are a few minor differences between this letter and the letter that will be mailed with the paper survey to sample members from the NAP participant list. Because sample members from the NAP participant list will receive an initial invitation to complete a web-based version of the survey, the letter that accompanies the mail survey will not be the first piece of correspondence they receive. They will receive the initial web invitation letter and a reminder postcard before the mail survey, so the letter that accompanies the mail survey references these prior mailings.

The letter that accompanies the area probability survey is the first and only letter these sample members will receive. To simplify pretest procedures, we chose to test this version of the cover letter. After completing the think-aloud pretest interview and debriefing with the interviewer, participants were given a copy of the invitation letter. The interviewer asked each participant to read the letter aloud and ask any questions. Participants said the letter was clear, and they understood what it was asking them to do. One suggestion offered was to use a different Spanish phrase for postage-paid envelope.

Food Security Status and Well-Being of Nutrition Assistance Program Participants in Puerto Rico, Appendix O. O-7
Pretest Methods and Summary of Findings