Appendix L.1 Pretest Methods and Summary of Findings

Appendix L.1
Pretest Methods and Summary of Findings

P.O. Box 2393
Princeton, NJ 08543-2393
Telephone (609) 799-3535
Fax (609) 799-0005
www.mathematica-mpr.com

MEMORANDUM

TO:

Insight Policy Research

FROM:

Betsy Santos

SUBJECT:

Survey Pretest Results for the Study of WIC Food Package
Costs and Cost Containment

DATE:

03/09/2016

Mathematica Policy Research conducted a pre-test of two instruments for the Study of WIC
Food Package Costs and Containment: (1) Participant Survey and (2) Former Participant Survey.
This memo summarizes the results from the survey pretest and summarizes the proposed
instrument changes based on these results.
The purpose of the pretest was to assess:
•

Survey length when administered in English

•

Survey flow

•

Respondents’ interpretation of questions

•

Respondents’ ability to recall required information

•

Completeness of response category options

To obtain timing estimates, the pretest interviews were conducted by telephone using
hardcopies of both surveys. At the end of the interviews we conducted a short debriefing to learn
whether respondents interpreted questions as intended. If respondents found any questions
difficult to answer, we asked why those questions were difficult and how easy or difficult it was
to recall the required information.
Testing Details and Procedures

We conducted 5 pretest interviews in English with the Survey of WIC Participants and 4
pretest interviews with the Survey of Former WIC Participants. A survey associate and survey
specialist at Mathematica conducted the 9 pretest interviews between February 15 and February
25, 2016.
Mathematica’s budget assumed that the Survey of WIC Participants would take 30 minutes
to complete on average and that the Survey of Former WIC Participants would take 20 minutes
to complete on average.
Respondents were sent a check for $30 for participating in the pretest.
An Affirmative Action/Equal Opportunity Employer

MEMO TO: Insight
FROM:
Betsy Santos
DATE:
03/09/2016
PAGE:
2
Recruitment and Respondent Profile

In February, Insight Policy Research contacted WIC state offices in Oregon and Florida to
explain the purpose of the pretest and request one list of current participants and one list of
former participants.
Respondents for the pretest were recruited from the sample files provided by the two WIC
state offices. Mathematica staff called potential respondents, explained the purpose of the pretest,
and scheduled a convenient time for the interview with those who agreed to participate. During
the pretest effort, however, many of the people we scheduled for the interview did not answer the
phone at the time of their scheduled interviews (4 current participants and 2 former participants).
In order to decrease the likelihood of getting a “no-show”, we revised the procedures and
attempted to complete the interview at the time of the recruitment call rather than scheduling an
appointment for a later time. This strategy was more successful. Nonetheless, the recruitment
effort was challenging overall, particularly for the former WIC participants. Out of the 30 phone
numbers in the sample file for the former participants, there were 7 (23%) phone numbers did
not work, and 5 of those were from Florida. Eleven cases did not answer, and 4 cases refused to
be interviewed. Moreover, 4 of the sample members in the Florida file only spoke Spanish.
Given these challenges with the Florida file, we were only able to complete 2 of the 9 interviews
with Florida participants.
We made an effort to recruit various types of respondent to ensure the different instrument
paths and fills were tested. These included households with (1) only a pregnant woman was
receiving benefits, (2) both the mother and child(ren) were receiving benefits, and (3) only one
or more child was receiving WIC benefits. Table 1 summarizes the characteristics of the
individuals who participated in the pretest interviews.
Table 1. Respondent Characteristics

Benefit Type

Gender
State

Current

Former

Total

Pregnant woman only receiving benefit

1

1

2

Woman and child(ren) receiving benefits

2

0

2

Only child(ren) receiving benefits

2

3

5

Male
Female

5

1
3

1
8

Oregon
Florida

3
2

4
0

7
2

MEMO TO: Insight
FROM:
Betsy Santos
DATE:
03/09/2016
PAGE:
3
Findings and Suggested Instrument Revisions
Survey of WIC Participants.
Survey Timing. The Survey of WIC Participants averaged 38.6 minutes. This is 29% longer
than the estimated time of 30 minutes. Section A, where the respondents had to recall their
satisfaction of the available brands and packages sizes of food items that they bought in the
previous month, was especially time consuming. Section C was also lengthy, particularly, where
the respondents had to recall whether or not they bought all, some or none of the foods they
could buy in the previous month and their suggestions for food items that they would like to buy
but cannot. The survey length in minutes by section is shown in Table 2.
Table 2. Survey Length in Minutes by Section
Section

R1

R2

R3

R4

R5

Intro/Screener

1

1

1

1

1

Section A

12

10

11

10

10

Section B

0

0

0

0

0

Section C

10

15

12

12

16

Section D

5

6

5

5

9

Section E

1

1

3

1

2

Section F

3

2

3

4

3

Section G

4

3

3

3

3

Total time

36

38

38

36

45

In order to shorten the time it takes respondents to complete the survey we have proposed
some changes and cuts to the survey to help decrease the completion time. Table 3 displays
proposed changes that will reduce respondent burden and bring the survey closer to the target
response time of 30 minutes.

MEMO TO: Insight
FROM:
Betsy Santos
DATE:
03/09/2016
PAGE:
4
Table 3. Proposed Modifications to Address Issues and Reduce Administration Time of the Survey
of WIC Participants
Question A5

Issue

Recommendation

Question A5

The list of difficulties in A5 are similar to the problems/difficulties mentioned in
A6, A7, and A8, especially items listed in the 2nd and 4th row of A5. It is
redundant to ask all of these questions.
Drop A5 completely to reduce survey time, and keep A6-A8 to capture
difficulties. Although our preference is to cut A5 entirely, another alternative is
to cut the 2nd and 4th items (finding foods in store and knowing which
package sizes add up to right amount) because they are very similar to A8
and A6 respectively. Also, the fourth item should be reworded to say “Knowing
which package sizes you can buy” as respondents had trouble understanding
the wording with the original phrasing.*

Issue

The fourth item in A5 is “Knowing which package sizes add up to the amount
you can buy” and respondents had trouble understanding the question.

Recommendation

Reword to “Knowing which package sizes you can buy.”*

Questions D1 and
D2/D12 (previously
D13)

Issue
Recommendation

It takes a long time for the interviewer to read all of the answer categories.
Switch the order of questions so that D2 is now D1. The interviewer can ask
the name of the store and if the interviewer is familiar with the name of the
store then they can code the store type on their own in D2. If the interviewer is
not familiar with the store type then they can ask the respondent what type of
store it is. Revise D2 and D12 to have an open ended response. The existing
list of response categories will be maintained in ALL CAPS so that
interviewers can code the answer by selecting from the list.*

Questions D6/D16

Issue

Respondents had difficulty answering because, while they could walk to the
place where they buy food, there were often many reasons why they did not
(e.g. dangerous, physical barriers, pregnant, etc.)
Drop this question. D7/D17 and D8/D18 capture distance. And other questions
in this section capture burden.

Recommendation
Questions E3/E4

Issue

Recommendation

Respondents had difficulty answering if they knew anyone who could get WIC
but didn’t because they didn’t know if others were eligible. One respondent
said that she knew someone who put off getting WIC but when asked E4 she
could not say why the person put off getting WIC.
Drop this question. We are not likely to get anything useful because
respondents are asked to provide second-hand information.
The questions in Question F1 and F4 have the same question stem.

Question F1/F4 (now
merged into Question
F1)

Issue
Recommendation

Question F3

Issue

Interviewers will need to provide definitions of most of the special diets most of
the time since many people are not familiar with the diet type.

Recommendation

We recommend dropping this question because of the time it will take for
interviewers to describe each of the special diets.

Issue

Most respondents said “yes” to these questions because they try to generally
eat healthy, rather than for any particular health reason.
We recommend dropping this question and rely on F1 and F4 to capture
special diets.

Question F2

Recommendation

We recommend combining both of these questions so that the question stem
can be read once.

*These changes were made to the Survey of Former WIC Participants in order to be consistent with the changes that we
made to the Survey of WIC Participants.

MEMO TO: Insight
FROM:
Betsy Santos
DATE:
03/09/2016
PAGE:
5
Survey Content. Overall, respondents seemed to understand most questions and did not
appear to have much difficulty answering them. Some questions, however, did required further
clarification. As such, proposed changes were made to question wording and question logic to
address any respondent misunderstandings of question intent. Table 4 provides a list of questions
where there was some issue during administration, and the suggested modifications.
Table 4. Current Participant Survey Content – Problems Identified and Recommended Changes
Screener section

Issue

Recommendation

1b

Issue
Recommendation

1b

Issue

Recommendation
A1

A1p

A respondent said that she was confused when she was asked
to confirm if she and her child received WIC foods. She said
that her child did not eat WIC foods, but the child’s name was
on the WIC EBT card.
Change the questions to ask the respondent to confirm who in
the household is receiving WIC benefits instead of foods.
The wording “I’d like to know the types of foods that WIC
provided…” may be interpreted as food that the respondent is
“getting” from WIC rather than what the respondent is “buying”
with WIC.

Recommendation

Change question wording so that it is clearly asking
respondents what they could buy with WIC for the previous
month.

Issue

The respondent said that she cannot buy canned fish until after
she has a child. There is no response category for this answer.
Add a “Not applicable” option for this question.

Issue
Recommendation

A1r

Since this is no longer the first question the phrase, “I’d like to
begin by confirming some information” is incorrect.
Omit the phrase “I’d like to begin by confirm some information.”

Issue

Recommendation
A1q and A1r

One respondent said that she just started receiving WIC
benefits at the end of January. This meant the interviewer could
not ask her about her food buying experiences in January.
Even though we expect to receive a sample file with only
participants who have been on WIC for more than one month,
we suggest adding a screener question (1a) that asks
respondents if they received WIC benefits in the previous 2
months. If they did not then they would skip out of the interview.
This would help account for any errors in the sample file with
benefit dates.

Issue
Recommendation

Respondent commented that they had a 1 month year old and
that she couldn’t buy the infant food yet.
The flag did not differentiate between infant ages 0-5 vs. 6-12
months. Since infants age 0-5 months cannot have cereal or
baby food, respondents with these infants should not receive
the question. We suggest creating two flags: Infant 1 = 0-5 and
Infant 2 = 6-12 so that this question skip can be programmed.*
Respondent was confused with “baby food fruits and
vegetables.”
Change to “jars of baby food.” Baby foods with meats is
allowed and so it is acceptable to keep this category broad.

MEMO TO: Insight
FROM:
Betsy Santos
DATE:
03/09/2016
PAGE:
6
A1b

Issue
Recommendation

A2b/A3b

Issue

Recommendation

A1_2

Issue

Recommendation
A3

Issue

Recommendation
A4

Issue

Recommendation
A7

Issue
Recommendation

B1

Issue

Recommendation

C2

Issue
Recommendation

Respondents did not understand what was included in our
definition of “grain.”
Reword the question to list the grains. It should say: Did you
buy bread, tortillas, pasta, or brown rice with WIC last month?
Respondents said that she didn’t know if she would be satisfied
with the brands or packaging of lactose free milk because she
has never purchased this food item.
Since WIC participants sometimes have a choice of what food
item to buy we recommend adding A1_1 and A1_3 and revising
A1_2. These questions ask the respondent to specify what type
of milk, grain and protein they chose to buy during the previous
month. Questions A2 and A3 will be asked only for the types of
milk, grain, and legume purchased by the respondent and not
for all types of milk, grains, and legumes available for purchase.
A1_2 (previously A1b) asked which grain the respondent
bought last month. Respondents did not understand what was
meant by “grain.”
Reword the question to list the grains we are asking about.
In the former participant survey respondents got confused
transitioning from A2, which asks about brands, to A3, which
asks about packaging. During A3 the respondents asked the
interviewer if the question was still referring to brands.
Add transition sentence to A3 so that it is clear that the
respondent should think about packaging.
A respondent commented that sometimes it is easy to shop for
WIC foods and sometimes it is difficult, but it depends on the
store.
Add response option, “Sometimes easy and sometimes
difficult.”*
Two respondents said that they have received WIC benefits for
less than 6 months.
Omit the words “in the past 6 months.”
Most respondents already mentioned that they had bought food
in January in questions A1_1, A1_2, and A1_3 and so it was
redundant to ask this question.
Add skip logic so that if the respondent indicates in A1_1,
A1_2, or A1_3 that they bought a food item then they would not
receive B1 and skip to C1.
Respondents said that they did not buy the item because they
chose the alternative for the month.
Add response option “Could not buy because chose alternative
item.”
Note: A1 asks “…could you buy [FOOD CATEGORY]” and in
section C we ask “did you buy” and reasons why they did not
buy. Therefore, we cannot use skip logic to skip over a question
when they buy an alternative.

MEMO TO: Insight
FROM:
Betsy Santos
DATE:
03/09/2016
PAGE:
7
C2 and C4

Issue

Recommendation

Intro to C3

Issue
Recommendation

C3 and C4

Issue
Recommendation

C4

Issue

Recommendation
C5a, C5c,C5d, C5e1, C5e2, C5f,
C5i, C5j, and C5q

Issue
Recommendation

C6d, C6e, C6f, C6i, C6j, C6l, C6q,
and C6r

Issue

Recommendation

C5e

Issue
Recommendation

D6 (previously D7) and D15
(previously D17)

Issue
Recommendation

One respondent indicated that there was no reason why she
did not purchase all of the milk in the previous month. When
probed by giving some possible reasons, she was able to
answer.
Add a response category for “no reason” and add a follow up
question after “no reason” is selected so that the interviewer
can read some possible responses to the question. Having this
probe as a separate question allows us to track how often a
probe was needed, and whether reading the responses had an
impact on response given.
The introduction did not read clearly for the scenario where one
child is receiving benefits
Add “your” before “X children.” Add “is a” as a fill so that the
sentence will read “…your 1 child is a” WIC participant.
The fill did not account for a scenario in which a woman only is
receiving the WIC benefits.
Add “you” as a fill option.
Respondent should not receive this question if they said in C2
that they “could not buy (the item) because (they) chose
alternative item.”
Add skip logic.
The interviewer reads the answer options and the “other” option
is awkward when read aloud.
Change the “other” option to “or something else?”
Referencing the “WIC food list” in the question stem resulted in
these respondents taking a long time to answer these
questions. Respondents had trouble remembering what foods
are on the WIC food list. One respondent had her WIC food list
and read through it for each question to make sure that she
was giving a food that wasn’t on the list. Moreover, respondents
were providing vague statements such as “all of the good stuff”.
Change the wording of the question to: Are there any specific
types or brands of (FOOD) that you would like to buy with WIC
but can’t?
Adding “specific types or brands of” helps focus respondents,
eliminating the vague responses. Removing “WIC food list”
makes the respondent less likely to depend on their WIC list to
respond. This will reduce the time it takes to answer these
questions.
Skip logic indicated that if the respondent answered “cold” then
the respondent should go to C9c2.
Updated logic so that if the respondent answers, “hot” then they
should be asked C5e2. If the respondents answers “cold” or
“both” then the respondent should continue to C5e1.
The respondent had difficulties providing the miles or blocks.
The respondent only provided the time.
Add interviewer probe: Your best estimate is fine.

MEMO TO: Insight
FROM:
Betsy Santos
DATE:
03/09/2016
PAGE:
8
D8 (previously D9) and D17
(previously D19)

Issue
Recommendation

D4

Issue
Recommendation

D13 (previously D14)

Issue
Recommendation

D18 (previously D20c)

Issue
Recommendation

D18c (previously D20c)

Issue
Recommendation

E1

Issue
Recommendation

E2

E3 (previously E4)

A respondent said that she goes to the store because “there
aren’t any other options because she lives in a small town.”
Add response option: No other options/Only store available.
Respondents said that they go to Trader Joes because there
are healthier options and food items with less preservatives.
Add response option: Healthier foods/organic foods*
Respondent commented that she uses a Walmart shopping
app to shop for WIC foods.
Add a question to ask the respondent to specify the name of
the shopping app that they use to buy WIC foods. As part of the
data analysis, these names can be matched with a list of
shopping apps specifically for shopping for WIC foods rather
than a general shopping app.
Respondents commented that they did not own a smartphone
and therefore it was difficult to answer the question.
Add response option: Don’t have a smartphone.
This option should not be read by the interviewer, but coded if
the respondents volunteers that he or she does not own a
smartphone.
The question needs the fill to be changed so that it accounts for
the plurality of “you” and “your family.”
Change the fill to be “you have/your family has.”

Issue

Answer categories do not apply to why the respondent did not
buy a WIC item. They are answer categories for why the
respondent did not go to the WIC store.

Recommendation

Recommend changing answer categories to apply for why
respondent did not buy item.

Issue

Respondents did not know what the eligibility requirements are
for WIC.
Add probe that clarifies that even if their child is no longer
eligible we still want to know if the respondent would apply
again.

Recommendation

F1 (previously F4)

The respondent could only say it takes “less than 10 minutes”
in the car to get to the store.
Add interviewer probe: Your best estimate is fine.

Issue
Recommendation

The skip logic after F4 is incorrect.
Change logic to say: IF “YES” TO “food allergy,” ASK F3. IF
“NO” TO “food allergy”, SKIP TO F4.

*These changes were made to the Survey of Former WIC Participants in order to be consistent with the
changes that we made to the Survey of WIC Participants.

MEMO TO: Insight
FROM:
Betsy Santos
DATE:
03/09/2016
PAGE:
9
Debriefings. Pretest respondents were asked five debriefing questions: (1) Did you find any
of the questions difficult to answer? If so, why were they difficult? (2) Did I ask you about
anything that was confusing or hard to understand? (3) How easy or difficult was it for you to
recall, or remember the information I was asking you about? (4) How confident did you feel
about your answers? (5) In general, is there anything you would change to improve the
questions?
Most respondents said that they didn’t find any questions confusing or hard to understand.
And they all expressed having a lot of confidence in their responses. When asked about recall,
one respondent said that since it was February 25th it was hard for her to recall the food items
she bought and ate in January. No respondents said that there was anything they would change to
improve the questions. However, a couple of respondents mentioned that they still used the
vouchers rather than the EBT card. Since we expect that all states will be using the EBT cards,
we did not suggest any revisions to the survey based on this feedback.
Next Steps for the Current Participant Survey. After implementing the proposed changes to
the questionnaire, we expect the Survey of WIC Participants to average between 35-37 minutes,
still longer than the 30 minutes that was budgeted. Administering the survey via CATI should
also streamline administration, but additional cuts are needed to ensure the survey stays within
30 minutes. We look to Insight and FNS for feedback on what additional cuts can be made to
reduce the survey’s overall length. Once the proposed changes are reviewed and the survey is
finalized, we will translate the surveys into Spanish.
Survey of Former WIC Participants
Survey Timing. The Survey of Former WIC Participants survey averaged 26.25 minutes.
This is longer than the estimated time of 20 minutes. However, this average time is skewed
because one respondent took 47 minutes to complete the interview. This respondent was
unusually talkative and therefore this interview is not an accurate measure of the survey’s timing.
If this interview is removed, then survey averaged 19.3 minutes, which is consistent with the
proposed estimate of respondent burden
Survey Content. Overall, respondents seem to understand most questions and did not appear
to have much difficulty answering them. Nonetheless, some questions required further
clarification. Table 5 provides a list of questions where there was some issue during
administration, and suggested modifications.

MEMO TO: Insight
FROM:
Betsy Santos
DATE:
03/09/2016
PAGE:
10
Table 5. Former Participant Survey Content – Problems Identified and Recommended Changes
A1c

Issue
Recommendation

Respondents were male and they were asked A2a.
Add question to the survey so that the interviewer codes the
respondent’s gender (male or female) based on their name
and speaking voice. If the interviewer is unsure of gender then
the interviewer will ask.

A2a

Issue

It does not make sense to ask this question if there is a male
respondent.
Add skip if the respondent is male.

Recommendation
B9, B11, and B12

Issue
Recommendation

The wording of these items are not in the same tense as B2B8.
Change wording of questions to match the tense of the
previous questions.

B16

Issue
Recommendation

Wording is inconsistent with the Current Participant survey.
Change wording to say “…and been sent back for a different
item…”

C1 (introduction)

Issue

A respondent had previously received WIC with older son
(before most recently receiving WIC with younger daughter)
and responded to some question thinking about both times.
Add text to the introduction that emphasizes that if the
respondent received WIC in the past then they should think
about their most recent enrollment.

Recommendation

C1, C2, C3

Issue
Recommendation

C2

Issue

Recommendation

The food categories are not consistent with the food
categories in the Current Participant Survey.
Add Lactose-free or lactose-reduced milk, dry beans, whole
wheat bread, whole wheat pasta, and brown rice. Change
“cow’s milk” to milk. Combine categories of hot and cold
breakfast cereal so that there is just one category of “breakfast
cereal.” Combine categories of frozen, canned, and bottled
juice so that there is just one category of “Juice.”
Respondents got confused transitioning from C1, which asks
about brands, to C2, which asks about packaging. During C2
the respondents asked the interviewer if the question was still
referring to brands.
Add transition text to C2 so that it is clear that the respondent
should think about packaging.

Debriefings. The debriefing questions were the same as the Survey of WIC Participants (see
above).
Most respondents said that they didn’t find any questions confusing or hard to understand
and they expressed having a lot of confidence in their responses. One respondent said that
repeating the question stem in Questions C1 and C2 was unnecessary and caused her to lose
focus. The interviewers will be trained to read the full stem initially for the first 2 to 3 items and
then just read the food category for the remaining items in the series. Similar to the Current

MEMO TO: Insight
FROM:
Betsy Santos
DATE:
03/09/2016
PAGE:
11
Participant survey, a couple respondents mentioned that they still used the vouchers rather than
the EBT card. Since we expect that all states will be using the EBT cards, we did not suggest any
revisions to the survey based on this feedback. All respondents said that it was easy for them to
recall information that was necessary to answer the survey.
Next Steps for the Survey of Former WIC Participants. Revisions are not needed to reduce
the burden of this survey. We do need to ensure that revisions made to the Survey of WIC
Participants are also implemented for this survey. Once the proposed changes are reviewed and
the survey is finalized, we will translate the surveys into Spanish.

cc: Nancy Cole and Rachel Sutton-Heisey