Appendix A_PreTest

Appendix A: Pre-Test

Sample and Recruitment

We pre-tested our draft survey instrument (see Exhibit A.1) with 6 persons in 3 research teams. The respondents were selected from the contact list (Appendix E). We stratified the contact list by the four affiliations through which research teams could have gained access to the FoodAPS-1 dataset. These strata included: 1) National Bureau of Economic Research (NBER) grantees, 2) University of Kentucky Center for Poverty Research (UKCPR) grantees, 3) cooperative agreement researchers, and 4) independent external researchers. We purposively selected one principal investigator per stratum to recruit for the pre-test.

For recruitment purposes, we communicated with potential participants via email (see Exhibit A.1). If we did not receive a response, we followed up via email within six business days (see Exhibit A.2). Of the four principal investigators we contacted, three agreed to complete the survey or to circulate the survey instrument within their research teams. Of the three research teams that completed the survey, one team also provided solicited feedback on the survey instrument itself. A total of 6 individuals participated in the pre-test.

Format

We offered participants the option of completing the survey either in writing (by completing the instrument in Word document format and returning it by email), in person, or via telephone. Of the three research teams that participated in the pre-test, two teams responded in writing, and one team provided responses both in person and via telephone. For the pre-test only, we included additional questions about the quality of the survey instrument and about completion time. The mean time reported under each mode was 30 minutes.

Pre-Test Survey Instrument

The initial instruments for the pre-test are reported in Exhibit A-3. The instruments include a mix of closed and open-ended questions. Topics include background information, data usage, and recommendations. We summarize revisions below.

Recruitment Script and Survey Revisions

All three principal investigators involved in the pre-test indicated a need to share the survey with collaborators who had more direct contact with the FoodAPS-1 data. Acknowledging this feedback, we revised our recruitment script and the survey instrument. For the recruitment script, revisions involved including language, asking principal investigators to identify their research team’s data analysts for us to contact directly. For the survey instrument, revisions involved including section headings that indicate whether survey sections contain general or specific questions.

The revised recruitment script (see Appendix B-1) indicates that principal investigators will identify the data analyst, who is better able to answer detailed questions. For example, researchers who directly analyzed FoodAPS-1 data may be better equipped to answer specific questions about data usage; whereas, principal investigators who had indirect contact with the data may be able to answer general questions. The revised survey instrument (see Appendix D) indicates whether survey sections are general or specific. This version of the survey instrument also allows respondents to indicate if another member of their research team will enter responses for a given section.

Other revisions we made based on pre-test feedback include: 1) reducing the telephone interview time burden estimate provided in the recruitment script from 40 to 30 minutes and 2) including an optional comments field after closed-end questions in the “Data Usage” section.

Exhibit A-1. Initial Recruitment Email for Pre-Test

Subject: A Questionnaire about FoodAPS User Experience (Pre-Test)

Dear [name of contact person]:

Please help us pre-test a questionnaire that we will use to help USDA learn from researchers' experiences with the National Household Food Acquisition and Purchase Survey (FoodAPS).

We have developed a draft questionnaire to assess the dataset’s uses, strengths, and needs for improvement. We will report results to USDA Economic Research Service to inform a possible next round of FoodAPS. 

 

Would you be willing to participate in this pre-test? Participation is voluntary and confidential and entails: 

• Completing the FoodAPS researcher questionnaire, either in writing (approximately 30 minutes) or via telephone (approximately 40 minutes)

• Answering a few short additional questions to help us improve our questionnaire. See questions a-d, appearing in the body of this email (approximately 7 minutes). 

Attached, please find the questionnaire and an informed consent form. If you are willing to complete the questionnaire, please review and return the informed consent form via email (on the consent form, a typed signature is sufficient).

 

Using your input in this pre-test, we will revise and improve the questionnaire and use it with a broader group of potential respondents. 

 

Please let us know if you have any questions or concerns. Then: (1) let us know by email within about 2 days (by Tuesday, 3/14) whether you will participate in this pre-test and (2) either return the questionnaire or let us know that we may contact you to schedule a telephone call within about 2 weeks (before Friday, 3/24).

Thank you for your consideration of this request!

1. How long did it take you to complete the questionnaire?

2. Were the questions easy or difficult to understand? Please highlight your response.

Easy		Neither easy nor difficult		Difficult
1	2	3	4	5

Was the questionnaire’s format easy or difficult to follow? Please highlight your response.

Easy		Neither easy nor difficult		Difficult
1	2	3	4	5

3. Were there any questions that you would add or remove from the questionnaire?

Exhibit A-2. Follow-up Recruitment Email for Pre-Test

(Sent electronically to potential pre-test participants if no response received within six business days of initial recruitment email)

Subject: A Questionnaire about FoodAPS User Experience (Pre-test)

Dear [name of contact person]:

I am writing to follow-up on a recent email about an invitation to pre-test a questionnaire on FoodAPS user experience. Would you be willing to participate in this pre-test?  

  

If yes, please send us a quick note by Wednesday, March 22 to confirm your participation. 

I am re-sending the informed consent form, questionnaire, and questionnaire feedback items. Please feel free to share with your collaborators if there are questions that might be better aligned with their user experience. 

Participation in this pre-test is voluntary and confidential. If you choose to participate, your input will help us refine our questionnaire before collecting feedback from a broader group of FoodAPS users. 

Thank you for your consideration of this request! 

Exhibit A-3. Draft Survey Questionnaire for Pre-Test

FoodAPS Users Questionnaire (Pre-test)

March 2017

Background

1. What motivated you and your research team to use data from the National Household Food Acquisition and Purchase Survey (FoodAPS)?

2. Which other datasets, if any, could you have used to answer your research question(s) if FoodAPS had not been available?

Data Usage

3. Please consider the documentation materials you used or are using for your study. How well did these materials meet your needs?

	Check or mark with X if used	Poorly 1	2	Neutral 3	4	Well 5	Comments (optional)
User’s Guide
Household-Level
Household Codebook
HH Geodata Codebook
Access Codebook
SNAP Eligibility Estimation Codebook
Individual-Level
Individual Codebook
Meals Codebook

FoodAPS Users Questionnaire (Pre-test)

March 2017

3. (Continued) Please consider the documentation materials you used or are using for your study. How well did these materials meet your needs?

	Check or mark with X if used	Poorly 1	2	Neutral 3	4	Well 5	Comments (optional)
Event and Item-Level
Places Codebook
FAH Events Codebook
FAH Items Codebook
FAH Item IRI Codebook
FAFH Supplementary Documentation
Event and Item-Level (continued)
FAFH Events Codebook
FAFH Items Codebook
Nutrient
FoodAPS Nutrient Coding Overview
FAH Nutrient Codebook
FAFH Nutrient Codebook
Geography Component
Retail Environment Study Codebook
Variable List: Retail Environment Study
Construction of Weekly Store-Level Food Basket Costs: Documentation

FoodAPS Users Questionnaire (Pre-test)

March 2017

3. Please consider the data files you used or are using for your study. How well did these data files meet your needs?

	Check or mark with X if used	Poorly 1	2		Neutral 3		4		Well 5		Comments (optional)
Household-Level Data
faps_household
faps_hhweights
faps_hhgeodata
faps_access
faps_snapelig_hh
faps_snapelig_unit
faps_snapelig_ind
Individual-Level Data
faps_individual
faps_reportstatus
faps_meals
Event and Item-Level Data
faps_places
faps_fahevent
faps_fahitem
faps_fahitem_iri
faps_fafhevent
faps_fafhitem
Nutrient Data
faps_fahnutrient
faps_fafhnutrient

FoodAPS Users Questionnaire (Pre-test)

March 2017

4. (Continued) Please consider the data files you used or are using for your study. How well did these data files meet your needs?

	Check or mark with X if used	Poorly 1	2	Neutral 3	4	Well 5	Comments (optional)
Geography Component Data
BGMain­_v6
BGRestaurant_v2
Tract_v5
County_v6
basketprices
basketprices_raw
Store Linker Files
PlaceID_IRI_TempERSID
PlaceID_TDLinx_TempERSID
PlaceID_TempERSID

User Experience

5. Compared to your experience using similar datasets, how long did it take you to orient yourself with FoodAPS? Please consider the time you took to review documentation, learn about the data enclave, and any other preliminary tasks. Please highlight your response or mark it with an X.

   Less time		Usual time		More time
   1	2	3	4	5

6. How would you rate the ease or difficulty of linking FoodAPS data files? Please highlight your response or mark it with an X.

Very easy		Neither easy nor difficult		Very difficult
1	2	3	4	5

FoodAPS Users Questionnaire (Pre-test)

March 2017

7. How would you rate the ease or difficulty of applying survey weights to FoodAPS data? Please highlight your response or mark it with an X.

Very easy		Neither easy nor difficult		Very difficult
1	2	3	4	5

8. What barriers, if any, did you encounter when using FoodAPS data?

Recommendations

9. What are your suggestions, if any, for adjustments to be made for a possible next round of FoodAPS? For example, your response may consider sample size, groups for oversampling, content focus areas, questions to ask, etc.

10. Are there any other comments you wish to share?

11. Would you be willing to answer follow-up questions via telephone? The purpose of a follow-up telephone call would be to clarify answers provided in writing. Please highlight your response or mark it with an X.

Yes

No