Appendix A: Pre-Test
Sample and Recruitment
We pre-tested our draft survey instrument (see Exhibit A.1) with 6 persons in 3 research teams. The respondents were selected from the contact list (Appendix E). We stratified the contact list by the four affiliations through which research teams could have gained access to the FoodAPS-1 dataset. These strata included: 1) National Bureau of Economic Research (NBER) grantees, 2) University of Kentucky Center for Poverty Research (UKCPR) grantees, 3) cooperative agreement researchers, and 4) independent external researchers. We purposively selected one principal investigator per stratum to recruit for the pre-test.
For recruitment purposes, we communicated with potential participants via email (see Exhibit A.1). If we did not receive a response, we followed up via email within six business days (see Exhibit A.2). Of the four principal investigators we contacted, three agreed to complete the survey or to circulate the survey instrument within their research teams. Of the three research teams that completed the survey, one team also provided solicited feedback on the survey instrument itself. A total of 6 individuals participated in the pre-test.
Format
We offered participants the option of completing the survey either in writing (by completing the instrument in Word document format and returning it by email), in person, or via telephone. Of the three research teams that participated in the pre-test, two teams responded in writing, and one team provided responses both in person and via telephone. For the pre-test only, we included additional questions about the quality of the survey instrument and about completion time. The mean time reported under each mode was 30 minutes.
Pre-Test Survey Instrument
The initial instruments for the pre-test are reported in Exhibit A-3. The instruments include a mix of closed and open-ended questions. Topics include background information, data usage, and recommendations. We summarize revisions below.
Recruitment Script and Survey Revisions
All three principal investigators involved in the pre-test indicated a need to share the survey with collaborators who had more direct contact with the FoodAPS-1 data. Acknowledging this feedback, we revised our recruitment script and the survey instrument. For the recruitment script, revisions involved including language, asking principal investigators to identify their research team’s data analysts for us to contact directly. For the survey instrument, revisions involved including section headings that indicate whether survey sections contain general or specific questions.
The revised recruitment script (see Appendix B-1) indicates that principal investigators will identify the data analyst, who is better able to answer detailed questions. For example, researchers who directly analyzed FoodAPS-1 data may be better equipped to answer specific questions about data usage; whereas, principal investigators who had indirect contact with the data may be able to answer general questions. The revised survey instrument (see Appendix D) indicates whether survey sections are general or specific. This version of the survey instrument also allows respondents to indicate if another member of their research team will enter responses for a given section.
Other revisions we made based on pre-test feedback include: 1) reducing the telephone interview time burden estimate provided in the recruitment script from 40 to 30 minutes and 2) including an optional comments field after closed-end questions in the “Data Usage” section.
Exhibit A-1. Initial Recruitment Email for Pre-Test
Subject: A Questionnaire about FoodAPS User Experience (Pre-Test)
Dear [name of contact person]:
Please help us pre-test a questionnaire that we will use to help USDA learn from researchers' experiences with the National Household Food Acquisition and Purchase Survey (FoodAPS).
We have developed a draft questionnaire to assess the dataset’s uses, strengths, and needs for improvement. We will report results to USDA Economic Research Service to inform a possible next round of FoodAPS.
Would you be willing to participate in this pre-test? Participation is voluntary and confidential and entails:
Completing the FoodAPS researcher questionnaire, either in writing (approximately 30 minutes) or via telephone (approximately 40 minutes)
Answering a few short additional questions to help us improve our questionnaire. See questions a-d, appearing in the body of this email (approximately 7 minutes).
Attached, please find the questionnaire and an informed consent form. If you are willing to complete the questionnaire, please review and return the informed consent form via email (on the consent form, a typed signature is sufficient).
Using your input in this pre-test, we will revise and improve the questionnaire and use it with a broader group of potential respondents.
Please let us know if you have any questions or concerns. Then: (1) let us know by email within about 2 days (by Tuesday, 3/14) whether you will participate in this pre-test and (2) either return the questionnaire or let us know that we may contact you to schedule a telephone call within about 2 weeks (before Friday, 3/24).
Thank you for your consideration of this request!
How long did it take you to complete the questionnaire?
Were the questions easy or difficult to understand? Please highlight your response.
Easy |
|
Neither easy nor difficult |
|
Difficult |
1 |
2 |
3 |
4 |
5 |
Was the questionnaire’s format easy or difficult to follow? Please highlight your response.
Easy |
|
Neither easy nor difficult |
|
Difficult |
1 |
2 |
3 |
4 |
5 |
Were there any questions that you would add or remove from the questionnaire?
Exhibit A-2. Follow-up Recruitment Email for Pre-Test
(Sent electronically to potential pre-test participants if no response received within six business days of initial recruitment email)
Subject: A Questionnaire about FoodAPS User Experience (Pre-test)
Dear [name of contact person]:
I am writing to follow-up on a recent email about an invitation to pre-test a questionnaire on FoodAPS user experience. Would you be willing to participate in this pre-test?
If yes, please send us a quick note by Wednesday, March 22 to confirm your participation.
I am re-sending the informed consent form, questionnaire, and questionnaire feedback items. Please feel free to share with your collaborators if there are questions that might be better aligned with their user experience.
Participation in this pre-test is voluntary and confidential. If you choose to participate, your input will help us refine our questionnaire before collecting feedback from a broader group of FoodAPS users.
Thank you for your consideration of this request!
Exhibit A-3. Draft Survey Questionnaire for Pre-Test
FoodAPS Users Questionnaire (Pre-test)
March 2017
Background
What motivated you and your research team to use data from the National Household Food Acquisition and Purchase Survey (FoodAPS)?
Which other datasets, if any, could you have used to answer your research question(s) if FoodAPS had not been available?
Data Usage
Please consider the documentation materials you used or are using for your study. How well did these materials meet your needs?
|
Check or mark with X if used |
Poorly 1 |
2 |
Neutral 3 |
4 |
Well 5 |
Comments (optional) |
User’s Guide |
|
|
|
|
|
|
|
Household-Level |
|
||||||
Household Codebook |
|
|
|
|
|
|
|
HH Geodata Codebook |
|
|
|
|
|
|
|
Access Codebook |
|
|
|
|
|
|
|
SNAP Eligibility Estimation Codebook |
|
|
|
|
|
|
|
Individual-Level |
|
||||||
Individual Codebook |
|
|
|
|
|
|
|
Meals Codebook |
|
|
|
|
|
|
|
FoodAPS Users Questionnaire (Pre-test)
March 2017
(Continued) Please consider the documentation materials you used or are using for your study. How well did these materials meet your needs?
|
Check or mark with X if used |
Poorly 1 |
2 |
Neutral 3 |
4 |
Well 5 |
Comments (optional) |
Event and Item-Level |
|
||||||
Places Codebook |
|
|
|
|
|
|
|
FAH Events Codebook |
|
|
|
|
|
|
|
FAH Items Codebook |
|
|
|
|
|
|
|
FAH Item IRI Codebook |
|
|
|
|
|
|
|
FAFH Supplementary Documentation |
|
|
|
|
|
|
|
Event and Item-Level (continued) |
|
||||||
FAFH Events Codebook |
|
|
|
|
|
|
|
FAFH Items Codebook |
|
|
|
|
|
|
|
Nutrient |
|
||||||
FoodAPS Nutrient Coding Overview |
|
|
|
|
|
|
|
FAH Nutrient Codebook |
|
|
|
|
|
|
|
FAFH Nutrient Codebook |
|
|
|
|
|
|
|
Geography Component |
|
||||||
Retail Environment Study Codebook |
|
|
|
|
|
|
|
Variable List: Retail Environment Study |
|
|
|
|
|
|
|
Construction of Weekly Store-Level Food Basket Costs: Documentation |
|
|
|
|
|
|
|
FoodAPS Users Questionnaire (Pre-test)
March 2017
Please consider the data files you used or are using for your study. How well did these data files meet your needs?
|
Check or mark with X if used |
Poorly 1 |
2 |
Neutral 3 |
4 |
Well 5 |
Comments (optional) |
||||
Household-Level Data |
|
||||||||||
faps_household |
|
|
|
|
|
|
|
||||
faps_hhweights |
|
|
|
|
|
|
|
||||
faps_hhgeodata |
|
|
|
|
|
|
|
||||
faps_access |
|
|
|
|
|
|
|
||||
faps_snapelig_hh |
|
|
|
|
|
|
|
||||
faps_snapelig_unit |
|
|
|
|
|
|
|
||||
faps_snapelig_ind |
|
|
|
|
|
|
|
||||
Individual-Level Data |
|
||||||||||
faps_individual |
|
|
|
|
|
|
|
||||
faps_reportstatus |
|
|
|
|
|
|
|
||||
faps_meals |
|
|
|
|
|
|
|
||||
Event and Item-Level Data |
|
||||||||||
faps_places |
|
|
|
|
|
|
|
||||
faps_fahevent |
|
|
|
|
|
|
|
||||
faps_fahitem |
|
|
|
|
|
|
|
||||
faps_fahitem_iri |
|
|
|
|
|
|
|
||||
faps_fafhevent |
|
|
|
|
|
|
|
||||
faps_fafhitem |
|
|
|
|
|
|
|
||||
Nutrient Data |
|
||||||||||
faps_fahnutrient |
|
|
|
|
|
|
|
||||
faps_fafhnutrient |
|
|
|
|
|
|
|
FoodAPS Users Questionnaire (Pre-test)
March 2017
(Continued) Please consider the data files you used or are using for your study. How well did these data files meet your needs?
|
Check or mark with X if used |
Poorly 1 |
2 |
Neutral 3 |
4 |
Well 5 |
Comments (optional) |
Geography Component Data |
|
|
|
|
|
|
|
BGMain_v6 |
|
|
|
|
|
|
|
BGRestaurant_v2 |
|
|
|
|
|
|
|
Tract_v5 |
|
|
|
|
|
|
|
County_v6 |
|
|
|
|
|
|
|
basketprices |
|
|
|
|
|
|
|
basketprices_raw |
|
|
|
|
|
|
|
Store Linker Files |
|
||||||
PlaceID_IRI_TempERSID |
|
|
|
|
|
|
|
PlaceID_TDLinx_TempERSID |
|
|
|
|
|
|
|
PlaceID_TempERSID |
|
|
|
|
|
|
|
User Experience
Compared to your experience using similar datasets, how long did it take you to orient yourself with FoodAPS? Please consider the time you took to review documentation, learn about the data enclave, and any other preliminary tasks. Please highlight your response or mark it with an X.
Less time |
|
Usual time |
|
More time |
1 |
2 |
3 |
4 |
5 |
|
|
|
|
|
How would you rate the ease or difficulty of linking FoodAPS data files? Please highlight your response or mark it with an X.
Very easy |
|
Neither easy nor difficult |
|
Very difficult |
1 |
2 |
3 |
4 |
5 |
FoodAPS Users Questionnaire (Pre-test)
March 2017
How would you rate the ease or difficulty of applying survey weights to FoodAPS data? Please highlight your response or mark it with an X.
Very easy |
|
Neither easy nor difficult |
|
Very difficult |
1 |
2 |
3 |
4 |
5 |
What barriers, if any, did you encounter when using FoodAPS data?
Recommendations
What are your suggestions, if any, for adjustments to be made for a possible next round of FoodAPS? For example, your response may consider sample size, groups for oversampling, content focus areas, questions to ask, etc.
Are there any other comments you wish to share?
Would you be willing to answer follow-up questions via telephone? The purpose of a follow-up telephone call would be to clarify answers provided in writing. Please highlight your response or mark it with an X.
Yes |
No |
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
Author | Mehreen Ismail |
File Modified | 0000-00-00 |
File Created | 2021-01-22 |