Appendix E Pretest Memorandum

Appendix E. Pretest Memorandum

Process and Use of the WIC Nutrition Assessment and Tailoring Study

November 15, 2019

This page is intentionally blank

Table of Contents

Chapter Page

Table of Contents i

1 Introduction and Methods 1

1.1 Pretest Respondents and Recruitment Methods 2

1.2 Data Collection 4

1.2.1 Cognitive Interviews 4

1.2.2 Testing of the WIC Participant Recruitment and Observation Processes and Forms 5

1.3 Data Analysis 5

2 Summary of Findings and Recommendations 1

2.1 Pretest of Local Agency Survey and Clinic Information Sheet 1

2.1.1 Local Agency Survey 2

2.1.2 Clinic Information Sheet 7

2.2 Field Test at WIC Clinics 9

2.2.1 Clinic Observation Form 10

2.2.2 Nutrition Services Observation Form 11

2.2.3 Identified Risks 14

2.2.4 Site Director Interview 15

2.2.5 Staff Interview 20

2.2.6 WIC Participant Screener 26

2.2.7 WIC Participant Interview 26

Table

1-1 Pretest respondents 2

Introduction and Methods

1

The U.S. Department of Agriculture’s Food and Nutrition Service (FNS) has contracted with Westat to conduct the WIC Nutrition Assessment and Tailoring Study (WIC NATS). The primary purpose of this study is to provide FNS with a comprehensive, detailed description of the WIC nutrition risk assessment process and describe the ways in which WIC clinics tailor participant benefits to address the results of the assessment. The study will collect a broad range of data from a large, diverse sample of 10 WIC State Agencies (SAs), all the Local Agencies (LAs) in these 10 SAs, and individual WIC sites where people apply for and receive WIC program benefits. We will collect the State WIC Plans and policy documents and State WIC management information system (MIS) data from these 10 SAs. We will also survey all LA directors within the 10 SAs about the policies and guidance that their sites must follow when conducting nutrition risk assessments and using the assessment results to tailor the services provided. We will collect extensive qualitative and quantitative information regarding WIC site level practices during visits to one purposively selected WIC site in each of 30 LAs. At each site, Westat will conduct interviews with the directors, staff who provide nutrition services, and participants, and will observe the clinic flow, WIC nutrition risk assessments, and the tailoring of program benefits based on the assessment results.

The data collected from this study will be used to address numerous research questions under four broad study objectives as follows:

1. Provide in-depth descriptive information on how a large, diverse sample of local WIC agencies performs the WIC nutrition risk assessment;

2. Systematically describe how a national sample of diverse local WIC agencies use the collection of nutrition assessment information to tailor program benefits, including food packages, nutrition education, breastfeeding promotion and support, and referrals to health and social services;

3. Investigate relationships between WIC nutrition services processes (including the nutrition risk assessment and the associated tailoring of program benefits), and the clinic experience, participant and staff perceptions, and overall clinic flow and efficiency; and

4. Analyze study findings to identify specific practices or features of nutrition service processes that facilitate the use of nutrition assessment information for providing tailored program benefits and are associated with participant and staff satisfaction.

In preparation for the main study, Westat conducted cognitive interviews and field tests of the data collection instruments and processes planned to collect information from the LA directors and at the WIC clinic sites. Findings from the pretest will ensure that the data collection instruments and procedures for the main study are feasible, and the forms, questions, and response choices are clear and yield the information necessary to answer the research questions. The Westat Institutional Review Board (IRB) reviewed all testing materials, including the recruitment and informed consent documents, prior to recruitment and data collection.

1.1 Pretest Respondents and Recruitment Methods

Westat selected a total of nine respondents for pretesting the LA director survey and the forms and interview guides for use at WIC clinics. Table 1.1 displays the number of respondents for each study instrument and number of observation locations for the pretest.

Table 1.1. Pretest respondents

Study Instrument(s)	Pretest Respondents/ Observation Location
Local Agency Survey Clinic Site Information Sheet	3 Local Agency Directors
Clinic Observation Form Nutrition Services Observation Form Identified Risks Data Collection Form	2 WIC Clinics
Site Director In-depth Interview Guide	2 WIC Clinic Site Directors
Staff In-depth Interview Guide	2 WIC Staff Who are Certified to Conduct Nutrition Risk Assessments
WIC Participant Screener WIC Participant Semi-quantitative Interview Guide	2 WIC Participants/Caregivers of Participants

LA Recruitment: Westat, with FNS approval, selected three SAs – Maryland, Virginia, and West Virginia to test the study methods and instruments. At the end of June 2019, FNS regional offices emailed letters of information to the State WIC directors in these three SAs. Westat followed-up with additional letters of information sent by email to the SA. This correspondence informed the SAs that Westat would be recruiting one of their LAs for the study pretest, summarized the study and pretest goals, and asked for their support with the pretest if their LA contacts them about the study. One week after contacting the SA directors, Westat staff emailed the LA directors from each of these SAs and conducted follow-up telephone calls as needed to recruit the pretest volunteers. Within two weeks, we successfully enrolled one LA director from each of the three States.¹ We asked each of these individuals to complete the LA Survey in hard-copy format and Clinic Site Information Sheet, and to document the time taken to complete the survey, and to describe their data sources used to complete the Clinic Site Information Sheet and record the time taken to complete the Sheet for each of their clinics. They were also instructed to highlight any wording that was not clear to them. Following receipt of the completed survey and spreadsheet, Westat team members scheduled a telephone-administered cognitive interview. To prepare for the WIC clinic visits and to tailor the Identified Risks Data Collection Form based on state-specific nutrition risk criteria, we also asked the Maryland and Virginia LA Directors to provide electronic copies or links to documents and tools used at their WIC sites that are related to the nutrition risk assessment and the most up-to-date list of the nutrition risk criteria that their WIC clinics use.

WIC Clinic and Staff Recruitment: For the field test of the WIC clinic data collection effort, we asked the recruited LA directors in Maryland and Virginia to recommend a clinic site for the pretest and send the clinic director an email to encourage them to participate. Westat emailed background information on the study to the recommended WIC clinic site directors, summarizing the purpose and planned activities for the one-day visit to their sites and requesting their voluntary participation and cooperation. Westat conducted one follow-up telephone call with each clinic director and both agreed to participate. Westat sent by email a sample schedule for the visit and a request for the client schedule for the day of the visit and conducted a final planning call within one week of the visit. The Westat team scheduled the visits to occur on days when the site directors anticipated least eight schedule WIC new enrollments and certification appointments. Prior to the site visit, the directors also agreed to participate in a cognitive interview and informally recruited one of their staff members who conducts nutrition risk assessments to participate in the pretest of the Staff Interview Guide.

Participant Recruitment: Participant recruitment occurred on-site at the WIC clinics. To assist Westat staff in identifying WIC participants who were potentially eligible for the main study, WIC clerical staff handed a study brochure (provided in advance by Westat to the WIC front desk staff) to WIC participants arriving for a WIC enrollment and/or certification visit. Westat staff were located in the waiting room and screened clients by completing the WIC Participant Screener to determine if the WIC participant was 18 years or older, whether the client was a new enrollee to WIC, and number of household members likely to be certified that day. At both sites, the first screened client was eligible and agreed to participate. One client and her baby were new to WIC and the second was a breastfeeding mother who had been on WIC during pregnancy and was returning to enroll her 8-week-old baby and certify herself as a breastfeeding postpartum WIC participant.

1.2 Data Collection

1.2.1 Cognitive Interviews

A trained survey methodologist conducted the cognitive interviews with LA directors by telephone. A trained interviewer, assisted by a note taker, conducted the cognitive interviews in-person with the WIC clinic director, staff and WIC participants. Each interview lasted 60 to 90 minutes each. Though the instruments varied in their structures, length, respondent type, and mode of data collection (i.e. telephone or in-person), each cognitive interview included the following steps²:

• The interviewer provided an introduction to the study, explained the study purpose, and read the respondent’s rights as a research subject.

• The LA directors provided verbal consent and all other interview respondents signed a written informed consent after the interviewer explained that participation was voluntary, responses would be confidential, and that the interview would be audio recorded if the respondent agreed.

• The interviewer administered the relevant interview guides and administered scripted probes in addition to asking for reactions to specific words and questions and responding to questions raised by the respondents.

• The interviewer and/or the note taker documented the issues, questions and clarifications needed as each respondent answered the interview guide questions and probes.

• At the conclusion of each interview section, the respondents had an opportunity to offer any additional feedback or reactions.

• After the end of the entire pretest interview, the interviewer and note taker thanked the respondent for participating.

As a key component of the WIC site director cognitive interview, we also tested questions and probes to guide the director in demonstrating key aspects of the SA’s MIS as it informs or helps the staff conduct the nutrition risk assessment and tailoring of benefits. At the conclusion of this interview, we also requested a copy of all tools that are used by staff at the WIC site to conduct the nutrition risk assessment (e.g. clinic flow forms or dietary questionnaires) so that we could refer to them in the staff interviews if needed and, if applicable, be sure to observe their use during the observation of the nutrition services components of the WIC certification or enrollment visit.

1.2.2 Testing of the WIC Participant Recruitment and Observation Processes and Forms

A site visit lead conducted the majority of the onsite interviews and a senior qualitative researcher acted as the note taker. The Westat site visit lead had a hard copy version of each of the forms tested, on which to record observations.

Using these documents, we tested each of these forms one time in each site. In addition to testing the content, the goals of this component of the pretest were to ensure that:

• WIC participants who are at the site for an enrollment or certification visit and staff who provide the nutrition service components of their visit could be easily screened for eligibility and recruited for the observation and interviews; and,

• that we could conduct the clinic and nutrition services observations smoothly without burden on the staff or WIC participants.

1.3 Data Analysis

Analysis of the telephone cognitive interviews and all the field test results included analyzing the notes from each interview and observation and a debriefing with the project director and other senior project team members to discuss the pretest findings. The team examined responses to identify questions that worked well and what could be improved, including where the respondents demonstrated confusion, hesitation, uncertainty, and/or discomfort and any aspects of the clinic flow and nutrition services visit that were difficult to document using the forms and within the timeframe of the visit. The team discussed the results of the analysis to validate the findings, confirm recommendations, and ensure that any changes would still allow for collecting the information needed to answer the study research questions. Finally, the study team determined the needed revisions to improve the accuracy of the data collected and strengthen the analysis of the quantitative and qualitative data.

The remainder of this report summarizes the general pretest findings on the ease of the data collection process, recruitment, and administration of each of the instruments, along with detailed notes on the issues identified in each tested instrument with recommendations for revisions that can address these issues.

Summary of Findings and Recommendations

2

2.1 Pretest of Local Agency Survey and Clinic Information Sheet

The recruitment of the LA directors for the pretest provided useful information on the readability, time to complete, and understandability of the items in the LA Survey and Clinic Site Information Sheet. On average, the respondents said that it took them 33 minutes³ to complete the LA survey and 20 minutes to complete the Clinic Site Information Sheet. Local agency directors also noted that in order to provide information on the percent of the caseload at each of their clinics that is high risk and the percent of the caseload with only one risk code (two questions included on the Clinic Site Information Sheet), they had to request this information from their SA. However, they also indicated that they received a response to this information request quickly.

2.1.1 Local Agency Survey

Pretested Question	Findings/ Recommendations	Revision
	Global Issues. None.	Question numbers match the revised instrument.
3. What forms (if any) are your WIC participants/parents or guardians asked to complete prior to appointments involving nutrition risk assessment? (Check all that apply.) Dietary intake forms for self and/or child Nutrition assessment questionnaires Forms to document participant’s eligibility (e.g. address, income, etc.) Other, specify:	No response option included for “no forms used” Recommendation: Add a gateway question similar to Q2 so that the question is only answered by those who ask participants to complete forms prior to appointments.	3. Are your WIC participants/parents or guardians asked to complete any forms prior to appointments that involve nutrition risk assessment? Yes No GO TO QUESTION 4 4. What kinds of forms are your WIC participants/parents or guardians asked to complete prior to appointments involving nutrition risk assessment? (Check all that apply.) Dietary intake forms for self and/or child Nutrition assessment questionnaires Forms to document participant’s eligibility (e.g. address, income, etc.) Other, specify: _____________________
4. What tools do staff utilize to conduct the nutrition risk assessment with a program participant? (Check all that apply.) Nutrition questionnaire Health questionnaire Medical information Diet history Measuring cups Baby bottles Other, specify:	Respondents noted that some of these tools listed are used in nutrition education, not nutrition assessment, and that it can be hard to differentiate between assessment and education sometimes. All respondents listed other tools they use during a nutrition risk assessment that were not on the list of response options. Some of these could be added to the list of response options. A respondent also noted that they use different terminology for the names of the different types of questionnaires. Recommendation: Change “tools” to “types of tools” to broaden the question. Add food models and drinking cups as response options. Add emphasis in bold to clarify that the question is only asking about the nutrition assessment component of the visit.	5. Which of the following types of tools do staff utilize to conduct the nutrition risk assessment component of the visit with a program participant? (Check all that apply.) Nutrition questionnaire Health questionnaire Medical information Diet history Measuring or drinking cups Food models Baby bottles Other, specify:
5. Which of the following types of staff typically perform the nutrition risk assessment at clinic site(s) operated by your Local Agency? (Check all that apply.) Registered dietitian/registered dietitian nutritionist (RD/RDN) Nutritionist (4 year degree/non-RD/RDN) Nurse (Registered Nurse (RN) or Licensed Practical Nurse (LPN)) Paraprofessional/nutrition assistant/nutrition aid Breastfeeding peer counselor Designated breastfeeding experts (including Certified Lactation Counselors, Certified Lactation Educators, and International Board Certified Lactation Consultants) Clerk/support staff Other, specify	Respondent differentiated between paraprofessional and nutrition aide; in her LA, aides are support staff that only do part of the NRA. Recommendation: separate paraprofessional from nutrition aide	6. Which of the following types of staff typically perform the nutrition risk assessment at clinic site(s) operated by your Local Agency? (Check all that apply.) Registered dietitian/registered dietitian nutritionist (RD/RDN) Nutritionist (4 year degree/non-RD/RDN) Nurse (Registered Nurse (RN) or Licensed Practical Nurse (LPN)) Paraprofessional Nutrition assistant/nutrition aide Breastfeeding peer counselor Designated breastfeeding experts (including Certified Lactation Counselors, Certified Lactation Educators, and International Board Certified Lactation Consultants) Clerk/support staff Other, specify
6. Think about the nutrition risk assessment trainings offered to all WIC staff who conduct nutrition risk assessments at your Local Agency’s clinics. Which of the following topics are included in the nutrition risk assessment training? (Check all that apply.)	Respondent indicated that some of the topics listed for this question are not necessarily included in training specific to the nutrition risk assessment (e.g. use of interpreter, multicultural awareness) Recommendation: Change the question to ask about all trainings for staff that conduct nutrition risk assessments.	7. Think about the nutrition risk assessment trainings offered to all WIC staff who conduct nutrition risk assessments at your Local Agency’s clinics. Which of the following topics are included in the trainings provided to these staff? (Check all that apply.)
8. How is the training conducted for each training topic you selected in Question 6? (Check all that apply.) (Only select answers for training topics you selected in Question 6.) [Note: For the hardcopy survey, this question was a grip with training topics as rows and training formats (e.g., national/State/Regional conferences) as the columns.]	Respondents noted that training format varies by level of staff. Because of this, respondents tended to check almost all formats for each training topic – that is, since training format differed by staff level, you can find almost all training topics in almost all types of formats. Recommendation: As is, the question does not result in much variation (if any) in format by training topic and the web version of this survey will result in respondents having to check all the same training formats for each topic separately, which is likely to be seen as burdensome. We recommend asking this question for training in general and not for each type of training.	9. What are the ways in which training is conducted for the training topics you selected in Question 6? (Check all that apply.) National/State/Regional Conference Training sessions/Courses at a State training center In-house training session (e.g., LA conferences, workshops) Clinic staff meetings State or local agency webinars Online training modules or courses Individual staff mentoring or coaching Other, specify ___________________
14. Now, think specifically about the breastfeeding assessment trainings offered to WIC staff who conduct breastfeeding assessments at your Local Agency’s clinics. Which of the following topics are included in the breastfeeding assessment training? (Check all that apply.) Grow and Glow in WIC Assessing mother’s personal beliefs and attitudes towards breastfeeding Assessing mother’s milk production/mother and baby’s dietary intake Assessing potential breastfeeding problems, breastfeeding techniques, and baby hunger cues Assessing participant breastfeeding goals Infant nutrition Motivational interviewing/counseling and participant-centered services skills Other, specify	Respondent stated that ‘Grow and Glow in WIC’ is a curriculum not a topic; it covers all the topics listed below it in the response options. Recommendation: Ask if they use Grow and Glow in WIC in a separate set of questions that mirror the current questions 14 – 17. Ask the current questions 14 – 17 for topics other than Grow and Glow in WIC.	15. Do you use the Grow and Glow in WIC curriculum for your breastfeeding assessment training? Yes, go to Q16 No, go to Q20 16. [If Q15 = yes] How often do WIC breastfeeding assessment staff receive training using the Grow and Glow in WIC curriculum? [Same response options as current question 15.] 17. [If Q15 = yes] How is the training conducted using the Grow and Glow in WIC curriculum? (Check all that apply.) [Same response options as previous question 16.] 18. [If Q15 = yes] Does your Local Agency offer refresher trainings using the Grow and Glow in WIC curriculum? [Same response options as previous question 17.] 19. What other topics not covered by Grow and Glow in WIC do you include in your breastfeeding assessment training? Other topics, specify: _________________ [If other topics listed, go to Q20; if no other topics listed go to Q24] Local Agencies that answered no to Q15, answer previous questions 14 – 17 with no changes to wording except for the removal of “Grow and Glow in WIC” from the response options. These questions are renumbered Q20 – Q23. Local Agencies that answered yes to Q15 and listed other topics in Q18 will answer Q21 – Q23 for each other topic listed.
24. How often do WIC staff at your clinics make internal referrals to the following specialists based on the results of a nutrition risk assessment? (Select one response for each type of referral.) Type of Internal Referrals Breastfeeding Peer Counselor WIC Designated Breastfeeding Expert WIC Registered Dietitian (RD)/ Registered Dietitian Nutritionist (RDN) Other, specify:	Respondents chose “always” for every type of referral because when these types of staff are available at their clinics, they always refer to them when needed. Recommendation: Re-write this question so that it does not focus on frequency. (Frequency of referrals will be obtained from MIS data and observations.) Instead, ask if referrals to each referral type is provided to all participants, only if indicated from the nutrition risk assessment results, or is not available.	31. For each of the following specialists, please indicate if an internal referral is offered … (Select one response for each type of internal referral.) To all WIC participants at my clinics Only when indicated based on the results of the nutrition risk assessment Specialist is not available in my clinics
25. How often do WIC clinic staff make referrals to the following health and social service programs or organizations based on the results of a nutrition risk assessment? (Select one response for each type of referral)	This question had the same findings as Q24 above. Also, respondents reported Emergency Food Provider and Head Start as programs to which they make external referrals. Recommendation: We recommend re-writing this question in the same manner as the question for internal referrals. Also, add Head Start and emergency food provider as response options.	32. For each of the following health and social service programs or organizations, please indicate if an external referral is offered … (Select one response for each type of external referral.) To all WIC participants at my clinics Only when indicated based on the results of the nutrition risk assessment Referral service not available to my clinics
30. How does the MIS at your clinics help staff tailor referrals offered to participants using information from the nutrition risk assessment? (Check all that apply.) Automatically tailors referrals offered to participants, but WIC staff can make changes Automatically tailors referrals offered to participants, and this cannot be changed by WIC staff Suggests possible ways to tailor referrals that WIC staff can choose from Does not help staff tailor referrals offered to participants Other, specify	Recommendation: Response edited for clarity based on recommendation of interviewer.	37. How does the MIS at your clinics help staff tailor referrals offered to participants using information from the nutrition risk assessment? (Check all that apply.) Automatically tailors referrals offered to participants, but WIC staff can make changes Automatically tailors referrals offered to participants, and this cannot be changed by WIC staff Suggests possible referrals to offer participants that WIC staff can choose from Does not help staff tailor referrals offered to participants Other, specify
37. Where are these clinic sites located? Primarily in urban areas Primarily in rural areas Primarily in suburban areas Clinics are located in a mix of urban/rural/suburban areas	One respondent answered both “primarily urban” and “primarily suburban.” Recommendation: Add this as a response option.	44. Where are these clinic sites located? Primarily in urban areas Primarily in rural areas Primarily in suburban areas Clinics are located in a mix of urban and suburban areas Clinics are located in a mix of urban/rural/suburban areas
38. Please list the three languages used most frequently by WIC participants in your Local Agency, and indicate the approximate percentage of WIC participants using each as their primary language. Language 1: _____________ Percent: ______ Language 2: _____________ Percent: ______ Language 3: _____________ Percent: ______	Respondent noted that she assumed we did not want to include English in the answer to this question, but question wording does not make this clear. Recommendation: Add wording to exclude English.	45. Please list the three languages other than English, used most frequently by WIC participants in your Local Agency, and indicate the approximate percentage of WIC participants using each as their primary language. Language 1: ________ Percent: ______ Language 2: _______ Percent: ______ Language 3: _______ Percent: ______

2.1.2 Clinic Site Information Sheet

Pretested Question	Findings/Recommendations	Revision
	Global Issues. None.
Urbanicity (Urban, Rural or Suburban)	Respondents found this confusing. Some clinics are classified as both urban and suburban. Recommendation: Based on FNS comments to the study plan, we will replace this with a request for the ZIP code for each clinic and use geocoding to determine urbanicity, which will provide us with a more uniform determination of this variable.	ZIP code
Primary Language Used by WIC Participants Served at This Clinic	The definition of primary language used is not clear. Recommendation: Add “spoken at home” to help define primary language.	Primary Language Spoken at Home by WIC Participants Served at This Clinic
Percentage of Participants at This Clinic with a High Risk Code <5% High risk 6-10% High risk 11-25% High risk More than 25% High risk	State agency assistance was required to run a report from the MIS for this information. In addition, because the spreadsheet contained dropdowns with defined ranges, the spreadsheet did not allow respondents to enter clinic-specific percentages. The actual range varies greatly among clinics and the defined categories were not always appropriate. Recommendation: Remove the drop downs allowing LAs to enter clinic-specific information. Because the data was easily obtained from the State Agency, keep the question, but also allow respondents to send the report they receive from the State Agency instead of entering the data.	Change to request from SA, delete here.
Percentage of Participants at This Clinic with Only One Documented Risk	State agency assistance was required to run a report from the MIS for this information. In addition, because the spreadsheet contained dropdowns with defined ranges, the spreadsheet did not allow respondents to enter clinic-specific percentages. Recommendation: Remove the drop downs allowing LAs to enter clinic-specific information. Because the data was easily obtained from the State Agency, keep the question, but also allow respondents to send the report they receive from the State Agency instead of entering the data.	Change to request from SA, delete here.

2.2 Field Test at WIC Clinics

Key findings regarding the flow and field-testing of the clinic level data collection process and activities are as follows:

• Westat staff were easily able to conduct all components of the field test during a one-day visit.

• Recruiting for observations and interviews and the conduct of the interviews at the WIC clinics worked very smoothly for all participant types and components of the field test. Recruited participants were interested in participating. The younger staff seemed nervous, according to the director at the first site, but after introductions at the beginning of the visit, this subsided to a degree. Making the staff feel comfortable and letting them know that we were not monitoring them was a key message and priority during our introductions and made a difference in the willingness of staff to participate.

• Meeting informally with the director to go over the updated schedule for the day and having a walkthrough of the site to see how a certification appointment flows through the office from time of arrival to departure gave us an opportunity to observe the waiting room environment before clients arrived. This process worked well and facilitated completion of the first part of the Clinic Observation Form, helped the day flow smoothly, and may have alleviated some nervousness on the part of the staff about our visit.

• At each of these sites, our process for identifying which WIC participants there for enrollment or certification visits worked very well and was not perceived as burdensome by the staff.  

• Length of visit varied. At the first clinic, the certification visit with the new mom who had been on WIC and the enrollment visit for the baby took 40 minutes total from time of arrival to departure and 25 minutes for the nutrition services components. The visit at the second clinic with a mother and baby who were both new enrollees and which used separate staff for completing the anthropometric measurements and blood work and for the rest of the nutrition assessment took a total of 2 hours and 10 minutes from the time of arrival to departure and 1 hour and 42 minutes for the nutrition services components.  

2.2.1 Clinic Observation Form

Pretested Question	Findings/Recommendations	Revision
	Global Issues. None.	Question numbers match the revised instrument.
General Site Facilities	Field testers indicated that it would be helpful to document whether or not the waiting room is for WIC and/or services for mothers and children only or for broader services. If used for a non-WIC clientele as well, the waiting room may not be child friendly or have WIC informational materials/videos. Recommendation: Add additional questions to document this.	2. Is the waiting room solely for the WIC clinic or shared with other programs? Only for WIC clinic GO TO Q4 Shared with other programs 3. What other programs share the space? Only for pregnant women, mothers and/or children Broader adult population Other, specify: ______________________
10. Educational materials available in waiting room	At one clinic, there were signs but no pamphlets or other handouts related to WIC; participants received those upon entering the area where WIC services were provided. There were some signs about WIC, but it was unclear to the data collector if signs count as “materials available.” Recommendation: Add description of the types of materials	12. Educational materials, related to nutrition and health, such as posters or pamphlets, available in waiting room
11. Toys/books/activities available to occupy children during appointments	This is covered on the Observation of Nutrition Services Components of WIC Certification form. Recommendation: Delete from this form.
12. Educational videos shown in waiting room	It was unclear to the data collector if we were specifically interested in education videos having to do with nutrition and health or if general educational videos should be counted here. Recommendation: Add description of the types of videos	15. Educational videos, related to nutrition and health, shown in waiting room
Observations of total time in clinic for participants with an enrollment or certification visit:	This proved to be difficult to track from the waiting room for enrollment and certification visits. The pre-test determined that better, more complete time data was obtained from the Observation of Nutrition Services Components of WIC Certification form. Recommendation: We recommend deleting the time tracking from this form. The visit observation form will provide total time for an expected 17 enrollment and certification visits per clinic, which will be sufficient for the analysis.	Remove time tracking from the Clinic Observation Form.

2.2.2 Nutrition Services Observation Form

Pretested Question/Section	Findings/Recommendations	Revision
	Global Issues. None.
Activities Conducted Table Footnote to table: Staff Type Options: Certified Professional Authority (CPA), RD/RDN, Lab technician, Breastfeeding Counselor or Coordinator, Paraprofessional Activities Conducted columns included: Nutrition risk assessment Nutrition education/counseling Food package prescription External referrals	There is some variation in the terms clinics use to describe staff. The Staff Type Options footnote has both staff types and credentials that are overlapping categories, e.g., some CPAs are RDs. At the Virginia (VA) clinic, they called the non-CPAs who do the anthropometric part of the visit a nutrition assistant. There are not “Activities Conducted” columns in the table for all visit activities. In VA, the nutrition assessment is conducted by two different people. Height/weight and lab work was done by a different person and in a separate area from the assessment. The table did not allow for entry of separate times for the two activities. Recommendation: Revise the first page of the form to be a landscape grid with 6 activity columns. Revise staff type examples in table footnote to be the same as the response options in the LA Director Survey (Q5) and Site Director Survey (Q2). Delete the “Duration” column as that can be calculated from the start and end times.	Footnote to table: Staff Type Options: Registered dietitian/registered dietitian nutritionist (RD/RDN), Nutritionist (4 year degree/non-RD/RDN), Nurse (Registered Nurse (RN) or Licensed Practical Nurse (LPN)), Paraprofessional, Nutrition assistant/nutrition aide, Breastfeeding peer counselor, Designated breastfeeding experts (including Certified Lactation Counselors, Certified Lactation Educators, and International Board Certified Lactation Consultants), and Clerk/support staff Activities Conducted columns included: Anthropomorphic measurements Laboratory measurements Nutrition assessment discussion Nutrition education/counseling Food package prescription Referrals Remove duration from table.
Which of the following best describes the seating arrangement and location of the computer?  WIC CPA had to turn away from participant to enter information in computer  Participant was seated across the desk or table from the WIC CPA, WIC staff faced the participant and could provide eye contact. Participant could not see the computer screen  There were no physical barriers (desk or computer) between the WIC CPA and participant, allowing for continued direct eye contact and participant ability to see the computer screen	The observer felt this question was difficult to answer because more than one option may be true. Also, the seating arrangement seems to be less of an issue than the extent to which the WIC staff person faced the participant during the visit and maintained eye contact. The “Participant Centered Communication Skills” section of this form already includes a question about maintaining eye contact. Recommendation: Delete this question in favor of the question about maintaining eye contact.	Remove this question from the observation form.
Were the participant’s height or length and weight measurements available to the CPA during this visit to the WIC site?  Yes, measurements were obtained on day of visit  Yes, recent measurements were obtained from an off-site health care provider  No measurements were available to the CPA at the time of the certification visit	It was not always possible to know this if the results were already in the participant’s file. Recommendation: Add a response option for “Could not be determined.”	Add Could not be determined to the response option list.
Were the results of a hemoglobin or hematocrit test available to the CPA during this visit to the WIC site?  Yes, hemoglobin or hematocrit test performed and value assessed on day of visit  Yes, hemoglobin or hematocrit test results were obtained from an off-site health care provider  No measurements were available to the CPA at the time of the certification visit	It was not always possible to know this if the results were already in the participant’s file. Recommendation: Add a response option for “Could not be determined.”	Add Could not be determined to the response option list.
During nutrition education, did the WIC staff specifically address any of the participant or caregiver’s nutrition risks?	This question assumes that the observer knows which nutrition risks were identified by the WIC staff person. The observer is not likely to know this until the staff person is asked to list the risks identified from the assessment in Q4 of the Identified Risks Data Collection Form. This form is completed after the assessment observation is finished. Recommendation: Delete this question.	Remove this question from the observation form.
If there were many nutrition risks discussed and needs expressed by the participant or caregiver, did the WIC staff help her/him select one or more priority needs or risks to work on?  Yes  No  Not applicable If yes, did the nutrition education provided during this visit address this identified priority?	The observer felt the phrase “did the WIC staff help her/him select” does not describe the discussion surrounding the issue of which of needs or risks to work on, making this question hard to answer. Recommendation: Combine these two questions and revise to clarify.	If there were many nutrition risks discussed and needs expressed by the participant or caregiver, how was the priority for topics for nutrition education/counseling determined?  Discussion with participant to determine her/his priorities  WIC staff identified the priorities using her/his judgment  No apparent prioritization of topics  Other, specify
During the visit, were any referrals made to external health or social service programs?	Recommendation: Edit question to better define “referral” and match the language used in the follow-up question.	During the visit, were any referrals made to specific external health or social service programs or organizations?
If yes, to what types of other health and social service programs or organizations was the participant or caregiver referred to for help?	In the LA survey cognitive interview, respondents suggested adding Head Start and Emergency food providers to the list of external referrals. Recommendation: Edit the list of external referrals to match the list on the LA survey.	(Mark all that apply)  Breastfeeding support  Childhood immunizations  Other Health services  Income support (TANF, SSI, UI)  SNAP/Food stamps  Medicaid/CHIP (we will determine what these programs are called in each state)  Substance use counseling  Domestic violence  Housing/Shelter  Child abuse prevention  Emergency food provider  Head Start  Other (Specify) _____________________
When using the MIS during the visit, to what extent did the WIC CPA try to use the MIS to build rapport with the participant/caregiver?	The observer could not determine how to answer this question given the visits observed. Recommendation: Edit to obtain information as to any use of the computer with the WIC participant.	Did the WIC CPA invite the participant to look at the computer screen for any purpose during the assessment?  Yes  No

2.2.3 Identified Risks

Pretested Question	Findings/Recommendations	Revision
	Global Issues. None	Question numbers match the revised instrument.
1. For research purposes can you tell us whether the participant you just served was new to the WIC program or has participated before for another infant or child?	The data collector was unclear on how to ask this question if more than one participant was observed? Recommendation: We will observe and record information on all participants who are being assessed. Therefore, we should document identified risk information for all participants.	1. For research purposes can you tell us whether the participants you just served were new to the WIC program or have participated before in a prior pregnancy or for another infant or child?
5. Did you decide that this participant was “high risk”?  Yes  No [If yes] Which of these risks made the participant “high risk”? (Mark all that apply on the Nutrition Risks Checklist below.)	Recommendation: Edit to allow for more than one participant observed.	5. Was this/Were any of the participant/s determined to be “high risk”? [If yes] Which of these risks made the participant “high risk”? (Mark all that apply on the Nutrition Risks Checklist below.)
6. Of all the nutrition risks identified for this participant, which ones: A) Were automatically generated by the Management Information System? B) Did you enter manually into the MIS? C) Are not included in the MIS record for this participant (for whatever reason)? (On the Nutrition Risks Checklist below, mark A, B, or C for each risk identified)	This question was re-written prior to the pretest based on pre-visit discussions with the WIC site directors. The revised question allowed us to determine if each identified risk was entered automatically by the MIS, manually, or not included in the MIS/computer record at all. Recommendation: Edit to allow for more than one participant observed and for clarity.	For each risk identified for these participants: 6. Was the risk: A. Automatically generated by the Management Information System (MIS)? B. Entered manually into the MIS? C. Not included in the MIS record for this participant (for whatever reason)? (On the Nutrition Risks Checklist below, enter A, B, or C for each risk identified)
7. Which of the nutrition risks identified by you were addressed by a referral to an external health or social service program or organization?	Question 7 does not include internal referrals. Recommendation: Add internal referrals and edit to follow the “For each risk identified for these participants:” introductory text that was added before Q6.	7. Was it addressed by a referral to an internal WIC staff member or to an external health or social service program or organization? (On the Nutrition Risks Checklist below, enter I (internal) and/or E (external) or N (neither) for each risk identified.)
8. Which nutrition risks identified by you were addressed in nutrition education and counseling?	Recommendation: Reword to group with Q6 and 7 and follow the “For each risk identified for these participants:” introductory text.	8. Was it addressed in nutrition education and counseling? (On the Nutrition Risks Checklist below, enter yes or no for each risk identified.)

2.2.4 Site Director Interview

Pretested Question	Findings/Recommendations	Revision
	Global Issues. None.	Question numbers match the revised instrument.
1. How many staff members are employed at this clinic? How many staff members perform nutrition risk assessments at this site?	Need to be sure that the directors include themselves. Recommendation: Edit question.	1. Including yourself, how many staff members are employed at this clinic? How many staff members perform nutrition risk assessments at this site?
2. What are the titles or roles of the staff performing nutrition risk assessments? Edited prior to pretest to: Which of the following types of staff perform any parts of nutrition risk assessments?	This question was edited prior to pretesting so it could be read as a multiple choice, mark all that apply question and to be clear that not all options are staff titles. Based on the pretest results, no further revision is needed.
3. Do staff use any tools to help them ensure that all aspects of the nutrition risk assessments are completed during the assessment process? a. Do staff have a protocol to follow that guides them through the assessment as it is being completed? [IF YES] What is the source of this protocol? [INTERVIEWER: COLLECT A COPY OF THE PROTOCOL] b. Does the Management Information System (MIS), or your WIC computer system, help staff ensure that a comprehensive assessment is completed in any way?	The lead question was interpreted in various ways by respondents. The follow up a and b questions are more specific and were well understood. Recommendation: Turn the lead question into an introduction to questions 3a and 3b.	We’d like to ask you about any tools staff use to help them ensure that all aspects of the nutrition risk assessments are completed during the assessment process.
4 a. Are there any other tools staff use to help perform the nutrition risk assessment? [INTERVIEWER: COLLECT THOSE TOOLS THAT WERE NOT COLLECTED FROM THE LOCAL AGENCY]	In response to this question, respondents listed computers, infant scales, and other medical equipment needed for the assessment. Recommendation: Edit to ensure that answers focus on written tools or documents.	4.a. Are there any other written tools or documents staff use to help perform the nutrition risk assessment? [INTERVIEWER: COLLECT THOSE TOOLS THAT WERE NOT COLLECTED FROM THE LOCAL AGENCY.]
Intro to 5: We would like to ask you some questions about your WIC Management Information System (MIS) or your computer system to find out how it helps staff document nutrition risks and tailor benefits in response to nutrition risk assessment findings. 5. Can you walk us through your MIS system and demonstrate how staff use it to document nutrition risks?	The words in the introduction and initial question are repetitive. Also, the initial question was not clearly understood by the respondent. This is the important first question to begin the walkthrough of how the staff use the MIS in the nutrition risk assessment so wording needs to be more specific to ensure all interviewers ask about the same aspect of the MIS. Recommendation: Create a new introduction that combines parts of the both the introduction and initial question. Revise the initial question to be more specific.	We would like to walk through some of the key functions of your MIS system to understand how it helps staff document nutrition risks and tailor benefits in response to nutrition risk assessment findings. Please show us how the MIS guides staff through the steps of the nutrition assessment process.
a. How does the MIS capture a client’s nutrition risks? Are they manually entered or automatically determined by the system based on the nutrition assessment information entered by the WIC CPA?	This question is missing the option where risks are determined automatically by the MIS, but additional risks can be manually added. Recommendation: Re-write this question to ensure all options are mentioned and include a follow-up so that respondents are asked to explain in what circumstances risks are manually added when the MIS generally determines the risks automatically.	Are nutrition risks always, sometimes, or never automatically determined by the MIS system based on the nutrition assessment information entered by WIC staff? [If answer is “sometimes”] In what circumstances are risks determined and manually entered by the WIC staff? Can you show us how that works?
b. How does the MIS support tracking changes in nutrition risks for participants over time?	They meaning of this question was unclear to respondents. They thought examples would be helpful. Recommendation: Add example.	b. How does the MIS support tracking changes in nutrition risks for participants over time? For example, does it automatically display a summary or key results from the previous visit or in some way provide a simple display of changes in the participant’s nutrition risks over time.
c. Can it generate reports or charts that show changes over time?	Respondent answered in terms of reports generated on no-show rates and caseload, not reports tracking changes for an individual over time. Recommendation: Edit for clarity.	c. Can it generate reports or charts that show changes over time for individual participants?
d. Does the MIS prompt staff to ask about and record updated participant risk information?	The meaning of this question was not clear to the respondent. Recommendation: Edit for clarity and to better match the research question.	e. Does the MIS prompt staff to ask the participant about previously recorded risk factors?
6. Does the MIS assist staff in tailoring WIC benefits? a. Can you show us how the MIS facilitates the tailoring of the food package? b. Does the MIS facilitate tailoring nutrition education and counseling? [IF YES] How is that done? c. Does the MIS facilitate tailoring of referrals? [IF YES] How is that done? [INTERVIEWER: ASK FOR A LIST OF BOTH INTERNAL AND EXTERNAL REFERRAL OPTIONS FROM THE MIS.]	Respondents understood the term tailoring, but felt the introductory question was duplicative of the follow-up questions. “WIC benefits” was narrowly interpreted by the respondents to mean the food package. Also, the wording of question 6a is inconsistent with 6b and 6c. Recommendation: Replace question 6 with introductory statement, and edit first question slightly to conform with the wording of the next two questions about the nutrition education and referrals.	6. We would like to see how the MIS helps staff tailor nutrition education and counseling, referrals, and the food package. a. Does the MIS facilitate the tailoring of the food package? [IF YES] How is that done? b.
	One site commented about aspects of their new MIS system that did not work well for conducting the nutrition services visit; respondents also offered suggestions for improvements. Recommendation: Add a question asking about functions of the MIS that do not work well.	6.d. Are there aspects of the MIS that do not work well for conducting the nutrition services visit with your WIC participants? [If yes,] What could be improved?
9. During the nutrition assessment, what is the process that staff follow for determining that a participant is high risk? a. Do they follow a written protocol that identifies the nutrition risk criteria that are considered high risk? [INTERVIEWER: IF YES, ASK FOR A COPY OF THIS DOCUMENT.]	At both clinics visited, the MIS automatically can identify if a woman is high risk. We were told however at one of these clinics that the staff can go in and identify a woman as high risk even if the system does not do so. If the protocol for determining high risk is programmed into the MIS, we would assume that this protocol is set by the SA. If the clinic is using a written protocol, we will not know the source unless we ask. Recommendation: Add a follow-up to question 9.a to ask the source of any written protocol. Add a question to find out if MIS does this automatically or if staff does it manually.	9.a. Do they follow a written protocol that identifies the nutrition risk criteria that are considered high risk? [IF YES] What is the source of this protocol? [INTERVIEWER: ASK FOR A COPY OF THIS DOCUMENT.] 9.b. Does the MIS system automatically identify when a client is high risk? Can the high risk designation only be made by the MIS or can staff determine that a certain risk or risks make a client high risk even if the system has not done so?
10.b Are participants referred to professional staff?	The meaning of the term “professional staff” was unclear to the respondents. At both clinics, high risk clients must be seen by a CPA, but it was not always the case that these staff had the credentials the respondents associated with “professional staff.” At one clinic, the CPA does not have to be an RD. At the second clinic, high risk participants are referred to the more experienced CPA who has an RD, but the other CPA on staff has an RD pending and is also considered “professional” staff by the director. Recommendation: Edit for clarity.	10.b Are high risk participants referred to specific staff? IF YES] What are titles and qualifications of the type of staff who see high risk participants at your clinic?
Next, I have some questions about how your clinic ensures staff exhibit cultural sensitivity when providing nutrition-related services. 11. Participants may speak different languages or have different cultural dietary preferences or observances. How does your clinic ensure that staff are sensitive to participants’ cultural and language differences when conducting nutrition risk assessments?	It was not clear to respondents how to answer the question that followed the introduction. One respondent focused only on language issues; the other spent a long time first explaining her definitions. Also, when questions 11 a, b, and c were changed from probes to individual questions, the introductory question became duplicative. Recommendation: To make the question flow better, revise the introductory question to ask the site director a lead-in question that asks him/her to list the different types of cultures and languages they serve at the site.	11. Participants may speak different languages or have different cultural dietary preferences or observances. What are the different languages that your participants speak and what cultural groups do you serve at this clinic?
Finally, I have a few questions about your clinic flow.	Respondents found it confusing to jump back and forth between the staff and participant perspectives in this section. Recommendation: Revise the introduction to the section by adding text to let the respondents know that first we will ask about staff and then will ask about participants.	Finally, I have a few questions about your clinic flow. First, let’s talk about it from your staff’s perspective. Then we’ll ask about clinic flow from the perspective of participants.
13. Do you think that your clinic flow works well for staff?	Respondent said no one would ever say ‘no’ to this question. Recommendation: Revise the question to ensure respondents think through how the clinic flow works for staff.	13. What does the clinic do to make the clinic flow work smoothly for staff?
	Interviewers suggest ending on a positive note by asking what works well at the clinic. This information would provide the site directors’ interpretation of promising practices that have made the process work better (Research question 4). Recommendation: Add a positive question to end of interview guide.	18. What are you most proud of about how the nutrition services process works at your clinic?

2.2.5 Staff Interview

Pretested Question	Findings/Recommendations	Revision
	Global Issues. None.	Question numbers match the revised instrument.
To start off, I want to ask you some questions about how you conduct a nutrition risk assessment.	The interviewers thought the interview started too abruptly. Recommendation: Add an introductory section that gives the WIC staff person some easy questions to answer and get them talking.	How many years have you been conducting nutrition assessments at this WIC clinic? __years What is your title at this WIC clinic? ____________ 3. Which of the following types of WIC clients do you see for WIC certifications All types of clients Only non-high risk clients Only high risk clients Other, specify
1. Please describe how you determine nutrition risk for a participant. a. Do you focus on desired health outcomes in assessing the WIC participant? b. Do you use referral information from health care providers when determining nutrition risk? What type of referral information do you use?	At both sites, determining nutrition risk is done by the computer. Staff enter information -- anthropometric, biochemical, dietary, and environmental -- and the system generates risks codes, which they can manually add to. However, most risks are automatically generated This question did not test well with one of the respondents. In particular, the phrase “desired health outcomes” was confusing. The respondent also asked, “what do you mean by ‘referral information?’” Recommendation: Reword questions a and b so that they are not yes/no questions, but require the staff person to consider the types of information they do use in the assessment process.	4. Please describe how you determine nutrition risk for a participant. a. What kind of information do you collect and examine so that you or your MIS can identify a participant’s nutrition risk or risks? b. What kinds of information provided by a health care provider do you take into account in determining a participant’s nutrition risks?
2. How do you ensure that all aspects of the nutrition risk assessment are completed when you are assessing a WIC participant? a. Do you use a checklist or other tool to help you complete a comprehensive assessment? b. Does the Management Information System (MIS), or your WIC computer system, facilitate ensuring the assessment is complete? If so, how?	Interviewers found this question to be awkward and felt it might have a negative connotation. Recommendation: Revise the question to make it more open-ended and remove any hint of a negative connotation.	5. Do you use any of the following to help you ensure that all aspects of the nutrition risk assessment are completed: A checklist or other tool? The MIS or your WIC computer system? Other items? Specify ____
3. What do you do if a participant provides unclear information during a nutrition risk assessment? a. How do you help the participant clarify unclear information?	Question 3a is redundant. It is just a different way of asking question 3. Recommendation: Remove question 3a and revise question 3 to better match the associated research question.	6. How do you clarify when relevant information from the participant is not clear?
Now, I’d like to ask you some questions about how you tailor a participant’s food package, nutrition education and counseling, nutrition and health goals, and referrals based on the results of the nutrition risk assessment. 5. Do you use information from the nutrition risk assessment to tailor food packages?  Yes → GO TO Q6  No→ GO TO Q7 6. [If the respondent answered yes to Q5] How do you use the information collected during the nutrition risk assessment to tailor food packages? a. What information is most important in deciding how (or whether) to tailor a food package?	Respondents at both sites made it clear that there is very little room for tailoring the food package—just choices for dairy and special formula related to medical conditions. There are a few rare circumstances where other substitutions can be made, for instance, for homeless persons who would get a different package since they cannot cook or store food in a refrigerator. Also, Q 6.a. did not make sense to one respondent, because felt the computer determines everything and that all information was important. Interviewer felt the gateway question (Q5) was not needed, lead to redundant discussion, and was more appropriate for a survey than a qualitative interview. Recommendations: Change the phrase “tailor food packages” to “make allowable changes or substitutions to the standard food package.” Remove the “gateway” question.	8. How do you use the information from the nutrition risk assessment to make allowable changes or substitutions to the standard food package? a. What information is most important in deciding how (or whether) to change the food package?
7. Do you modify nutrition education based on the findings of a participant’s nutrition risk assessment?  Yes → GO TO Q8  No → GO TO Q9 8. [If the respondent answered yes to Q7] How do you modify nutrition education based on the findings of a participant’s nutrition risk? a. Do you modify counseling methods? The delivery medium? b. How are the nutrition education topics you offer to each participant informed by their nutrition assessment?	Again, the gateway question (Q7) lead to redundant discussion. For question 8a, neither respondent indicated that they modify counseling methods or the delivery medium based on the nutrition risk and this question “fell flat.” Respondents thought question 8.b. was redundant with Q7. Recommendation: Combine questions 7 and 8 to reduce redundancy.	9. How do you modify nutrition education based on the findings of a participant’s nutrition risk assessment? Do you modify: Do you modify counseling methods? The delivery medium? Nutrition education topics offered? Other items?
9. When a participant has multiple nutrition risks, how do you prioritize nutrition education topics? a. Do you discuss all nutrition risks identified or do you select one or more risks for discussion?	The respondent from MD seemed not to initially understand this question. She mentioned that participant category would have more influence on education topics (i.e., breastfeeding mom) than nutrition risks. Recommendation: Revise wording to clarify.	10. When a participant has multiple nutrition risks, do you provide nutrition education for all of their risks or do you select certain risks for nutrition education sessions? a. [If select certain risks] How do you decide which risks to select?
10. Are secondary (or subsequent) nutrition education contacts tailored to the participant’s nutrition risk(s)?  Yes  No [GO TO Q11] a. If so, how?	The MD respondent had difficult with this question as well. The VA respondent noted that tailoring education has 3 aspects: 1) how soon they are asked to come in, 2) the format (e.g., one-on-one counseling or a class) and 3) the topics covered. Again, the gateway question lead to redundant discussion. Recommendation: Combine question 10 and 10a. Use the VA respondent’s 3 aspects of nutrition education tailoring in the question.	11. How, if at all, are secondary (or subsequent) nutrition education contacts tailored to the participant’s nutrition risk(s)? Do you tailor: How soon participants are asked to come in? The format (one-on-one sessions or classes)? The topics covered? Other items?
11. Do you use the nutrition assessment information to help guide participants in establishing nutrition and health goals?  Yes  No [GO TO Q12] a. If so, how is the assessment information used? b. Do you tie goals directly to the identified nutrition risks?	Respondents had difficulty with this question. Recommendation: Change this group of questions to a single open-ended question that avoids repetitive yes/no questions.	12. How do you use the information from the nutrition assessment to guide participants in establishing nutrition and health goals?
12. Does your clinic’s MIS, or computer system, contain information that you can use to help you tailor program benefits based on the results of a participant’s nutrition risk assessment?  Yes  No [GO TO Q12] a. What information in the MIS do you use to help you tailor program benefits?	Again, the gateway question lead to redundant discussion. Respondents’ answers did not cover all three components of “program benefits.” Recommendation: Change to open-ended question to avoid repetitive yes/no questions and add wording to clarify that “program benefits” is referring to education, the food package, and referrals.	13. What information does your clinic’s MIS display to help you tailor program benefits, including nutrition education, the food package and referrals?
14. Participants may speak different languages or have cultural dietary habits. How do you ensure that you are aware of cultural and linguistic differences when conducting the nutrition risk assessments? a. What do you do to be aware of cultural dietary habits or other dietary restrictions? b. What other cultural considerations have you taken into account when conducting nutritional risk assessments? c. How can nutrition risk assessments at your clinic do a better job of asking about and addressing important cultural factors? 15. How do you ensure that you are aware of cultural and linguistic differences when providing nutrition education? a. Do you use an on-site interpreter or language line service when providing education to a participant who speaks a different language than yourself? b. How do you recognize and support participants’ cultural dietary preferences or observances when providing nutrition education? c. What other cultural considerations have you taken into account when providing nutrition education?	Interviewers felt questions 14 and 15 needed an introductory question to get them thinking about the types of cultural and language differences they see in their clinic. Question 14 is double-barreled, asking about both linguistic and cultural differences. Answers will differ for these two types of differences. Question 14a is redundant with the initial question. Recommendations: Add a lead-in question that asks them about the cultural differences they encounter in their work at the clinic (similar to the question added to the site director interview guide.) Divide questions 14 and 15 into separate questions for language differences and for cultural dietary habits. Create a separate question for other cultural considerations.	15. What are the different languages that your participants speak and what cultural groups do you serve at this clinic? 16. What resources are available to assist you with non-English speaking participants, such as an interpreter or language line service, for… Conducting a nutrition assessment? Providing nutrition education? 17. For participants with specific cultural dietary habits, how do you … find out about them when you are conducting the nutrition risk assessments? Recognize and incorporate them when providing nutrition education? a. 18. Apart from linguistic and cultural dietary habits, what other cultural considerations have you taken into account when conducting nutritional risk assessments? How can nutrition risk assessments at your clinic do a better job of asking about and addressing important cultural factors?
17. Do you routinely follow-up on nutrition and health goals in the following situations: In the first visit after an assessment? When the participant is high-risk? When the visit is for nutrition education At the next certification visit? Are there other situations after which you would follow-up on nutrition and health goals? If so, please specify:	The first visit after an assessment was confusing for both MD and VA respondents. Recommendation: Remove this response option from the question.	20. Remove the “In the first visit after an assessment” response option.
18. Think about cases where you referred a participant to services based on information from their nutrition risk assessment. When do you follow-up on these referrals? In the first visit after a referral is made When the participant is high-risk When information has been received from the referral When information has not been received from the referral At the next certification visit Are there other situations when you would follow-up after referring a participant to services? If so, please specify:	The first visit after a referral is made was confusing for both MD and VA respondents. Recommendation: Remove this response option from the question.	21. Remove the “In the first visit after an assessment” response option.
30. We could make better use of technology to increase participant satisfaction and improve clinic flow.	This question is double-barreled, asking about both participant satisfaction and clinic flow. Recommendation: Focus this question on clinic flow since participant satisfaction is subjective.	33. We could make better use of technology to improve clinic flow.
33. What recommendations would you make for improving clinic flow for staff? For participants?	Respondent thought the question asked what recommendations they would make to others, not recommendations for how clinic flow can be improved in their own clinic. Recommendation: Re-word to clarify.	36. If you could make any changes you wished, what would you do to improve clinic flow in this clinic for staff? For participants?
	Interviewers suggest ending on a positive note, asking what works well at the clinic. Recommendation: Add question shown.	38. What do you think this WIC clinic does really well that other clinics could learn from?

2.2.6 WIC Participant Screener

Question number	Findings/Recommendations	Revision
	Global Issues. None.
1. Are you here today for an enrollment or re-certification visit?	Participants did not know these terms. However, they were able to say whether they are currently on WIC or are at the clinic to get enrolled. With front desk staff handing out study brochures to participants at the clinic for enrollment or re-certification visits, this question was not needed. Recommendation: Delete this question from the screener.	Remove question from screener.
2. If the person is accompanied by a child/children] Are both you and your child/children being certified today?	This question was added to the screener to help in the analysis of time data collected on the Clinic Observation Form. Recommendation: Because we have recommended not using the Clinic Observation Form to collect time data, this question is no longer needed. We recommend removing it.	Remove question from screener.
It looks like you qualify for the study. Can I tell you more about the study and you can decide whether you want to help us out? If you agree to help us, you’ll receive a $20 gift card in the mail.	Edited prior to the pretest to clarify when the person would receive the incentive. Recommendation: Revise to indicate that the gift care will be provided when the interview is completed.	It looks like you qualify for the study. Can I tell you more about the study and you can decide whether you want to help us out? If you agree to help us, you’ll receive a $20 gift card today if you have time to stay for a brief interview after your appointment or in the mail if you do the interview later by phone.

2.2.7 WIC Participant Interview

Question number	Findings/Recommendations	Revision
	Global Issues. Participants understood all questions other than those noted below. Questions for participants who don’t speak English well and the 5 questions about forms (questions 5 – 9) were not tested because the two participants selected for observation spoke English well and there were no WIC nutrition risk assessment-related forms that participants were asked to fill out before their visit.	Question numbers match the revised instrument.
What type of visit was today’s/date’s visit to the WIC clinic? Enrollment Re-certification	Participants are not familiar with these terms. Recommendation: Because this information is obtained on the visit observation form, the question is not needed here and we recommend deleting it.	Remove question from participant interview.
4. Were you given any forms to complete before the visit, either at home or in the clinic waiting room? Interviewer: Check all that apply. Yes, at home Yes, in the clinic waiting room No → GO TO Q10	Participant mentioned all forms she needed to complete, not just those related to health or nutrition. Recommendation: Edit for clarity	Before the WIC visit, either at home or in the clinic waiting area, were you given any forms to complete with questions about your or your child’s health or food habits? Interviewer: Check all that apply. Yes, at home Yes, in the clinic waiting room No → GO TO Q10
How would you rate your satisfaction with the talk you had with WIC staff about your/your child’s health and diet during your visit today [on date]? How would you rate your satisfaction with the nutrition education and counseling you received?	Participant thought the topics asked about in questions 26 and 27 were identical and could not distinguish between them. Recommendation: Q26 is about the nutrition assessment and Q27 is about nutrition education. Recommend revising Q26 to clarify that it is asking about the nutrition assessment. Remove “today [on date]” as all the questions address the current day’s visit, referenced at the beginning of the interview.	22. How would you rate your satisfaction with the WIC staff’s review of your nutrition status? That is, the height and weight measurements, blood tests, and questions about diet, health, and home environment.
I feel that the food choices provided by WIC help meet my [my child’s] needs. I feel that the food choices provided by WIC take my [my child’s] food preferences into account. How would you rate your satisfaction with the food choices you were given when discussing your food package at the WIC clinic?	Participant thought these questions were repetitive. Recommendation: Probes in Q28 provide more information; suggest deleting Q21 and 22 and using them as probes for Q28.	24. How would you rate your satisfaction with the food choices you were given when discussing your food package at the WIC clinic? Interviewer, have the participant explain their choice by asking each of the following questions: Why? What makes you say [e.g. very satisfied]? Do you feel that the food choices provided help meet your/your child’s needs? Do you feel that the food choices provided take your/your child’s preferences into account? What did you like most/least about the WIC foods you received? Is there something the clinic could have done to improve the WIC foods you received? [IF YES]What would make the WIC foods you receive more valuable to you?
25. The referrals I received during my WIC clinic visit were helpful to me. 29. [If participant received referrals – see Q19] How would you rate your satisfaction with the referrals you received to other services?	Participant thought these questions were repetitive. Recommendation: We recommend deleting Q25.	Remove Q25 from participant interview.
Do you feel satisfied with how well the services you received during your clinic visit reflected your specific needs and concerns?	Recommendation: Edit for clarity.	26. Do you feel satisfied with how well the WIC services you received during your clinic visit reflected your specific needs and concerns?

1 Two additional LA directors who were initially contacted said they would prefer not to participate at this time because of staff shortages and heavier summer workloads.

2 For the cognitive interviews with the LA Directors, each respondent was also asked for general feedback on the instrument and if the respondent had not written down the number of minutes it took her to complete the survey we asked her to estimate this time.

3 Two LA directors reported that completing the survey took 20 minutes; the third LA director said the survey took “less than one hour” to complete. These numbers were averaged to obtain the 33 minute burden estimate.