F Summary of Pretest Methods and Findings

To:	Jenny Genser
From:	SCANR Study Team
Date:	May 11, 2017
Subject:	Deliverable 2.2: Pretest of Data Collection Instruments for the SCANR Study (Revised)

The purpose of this memo is to present the findings from the nine pretests of the survey instruments for the SNAP SCANR study. We conducted one pretest with a scanning technology vendor for the industry interview guide, three pretests with small, SNAP-authorized retailers for the questionnaire for the SCANR Survey, and five pretests with small, SNAP-authorized retailers for the follow-up interview guide. For the SCANR Survey, we also had informal contacts with one industry expert and five trade associations who provided additional feedback.

We present below a taxonomy for front-end register systems that we developed based on information gathered during the pretest and that helped to inform some of the revisions we made to the instruments. The remainder of this memo contains three sections of pretest findings—one for each of the data collection components. Each section describes the pretest participants, the procedures followed for the pretests, lessons learned and recommendations for the full-scale study, recommended changes to the instruments, and participant burden estimates from the pretest findings.

taxonomy for front-end register systems

Based on findings from the pretest interviews with retailers, comments we received from several of the informal contacts with trade associations, and discussions with our expert consultant, Art Burger, we developed the taxonomy for front-end register systems shown in Figure 1. This taxonomy helps provide a framework for the cost analysis and a structure for collecting information on the types of configurations currently in place and the cost to install a new system versus make modular upgrades to an existing system to meet the minimum requirements of the Farm Bill (i.e., use scanner and include flags to verify whether products are SNAP eligible), as well as upgrading to a system that is integrated with the EBT payment terminal. Figure 1 shows that there are five possible system configurations, depending on whether the system has a scanner or not, has a UPC database with a SNAP flag or not, and whether the system is integrated with the EBT payment terminal or not. We also identified changes in terminology that were necessary, such as “scanning technology” versus “scanning system.” We revised all three sets of instruments to reflect these changes.

Figure 1. Taxonomy for Front-End Register Systems

pretest of interview guide for industry interviews

We conducted one pretest of the industry interview guide. The following sections provide a description of the pretest participant, the procedures followed for the pretest, and the lessons learned and recommendations for revisions to the industry interview guide.

Participant

To recruit the sole industry interview pretest participant, we consulted a list of vendors that conduct business in each state. Our consultant, Art Burger, created this list by searching online and using his knowledge of the industry. For convenience, the vendor that services the state of Louisiana was selected, because the RTI staff member who conducted the pretest lives in this state. The interviewer called the vendor, explained the project, and requested an interview for the following week.

Procedures

On the day of the interview, the interviewer sent the participant an email with the text from the introduction of the interview guide. This text served as a brief background for the participant and reminded him about the questions we would ask, the time commitment, and our confidentiality pledge. Sending it in advance saved time during our telephone interview. We began the phone call by asking him if he had any questions about the information sent by email and then proceeded with the questions in the order they appeared on the interview guide. The call lasted approximately 1 hour, which is consistent with the estimated burden.

We also made additional changes to the interview guide based on information obtained from the interviews with retailers to pretest the SCANR Survey questionnaire and follow-up interview guide. The changes made based on the industry interview pretest and this additional information are summarized below.

Lessons Learned and Recommended Changes to the Interview Guide

We found there were no concerns with the flow, timing, and information obtained during the industry interview pretest. The pretest participant sells hardware, software, and bundled systems, so we could ask questions from all sections of the interview guide.

In the introduction and throughout the questions in the interview guide, we changed the term “scanning hardware” or “scanning equipment” to “scanning technologies” to be consistent with the survey instrument.

In Section 1, we removed the second part of Question 1b that asked if the systems sold are resident, mobile, or cloud based after determining this information was not needed to address the research questions for the study. We also deleted Question 2 on number of registers or lanes, because it is redundant with a question asked in the SCANR Survey questionnaire.

In Section 2, we added clarifying instructions to Question 1, to include hardware components in addition to the scanner. We did not ask Questions 6 through 9 in the vendor pretest, which ask about service contracts and maintenance, insurance, and utilities costs, during the interview because these questions were not relevant. The scanners are inexpensive pieces of equipment (ranging from $30 to $300) and thus have no annual maintenance costs or service contracts associated with the equipment itself. The service contracts are for software or bundled systems, not the hardware. Moreover, we do not believe that it will be possible to isolate the additional energy costs of a scanning system from the store’s total energy bill. Therefore, we removed these questions from the hardware section of the interview guide.

In Section 3, we added new questions to determine the cost of an entirely new point-of-sale system that is integrated with the EBT payment terminal versus modular upgrades for an integrated system to verify SNAP-eligible products and to manage inventory. We also added questions to determine the costs for upgrades if not currently integrated with the EBT payment terminal. With these new questions, we deleted Questions 3 through 5 that asked about verifying SNAP-eligible products and inventory management.

We deleted all the questions in Section 6, which asked about integration with the payment terminal, because questions about the payment terminal are now asked in Section 3.

pretest of questionnaire for the scanr survey

Three small SNAP-authorized retailers officially pretested the instrument for the SCANR Survey, and an industry expert and four associations agreed to review the SCANR Survey instrument and provide feedback. The following sections discuss the procedures followed for the pretests with retailers and the procedures followed for our informal contacts with industry experts and trade associations. Next, we discuss the lessons learned, recommendations for the full-scale study, and our recommended changes to the survey instrument. We conclude with the burden estimate based on the pretest findings.

Procedures for Pretests with Retail Stores

To recruit participants, we used the preliminary sample frame to select stores located in the areas of Raleigh, North Carolina and Columbus, Ohio, where RTI staff are located who could potentially conduct in-person interviews. Additionally, to identify stores in rural areas we used a list of stores in rural North Carolina to contact for phone interviews. These lists included the owner’s name, store location, type (e.g., grocery, convenience store), and phone number. We contacted the store owner or manager by phone using a script to request their store’s participation in the pretest of the SCANR Survey by phone or an in-person interview. The FNS Contracting Officer’s Representative was invited to participate in the phone interviews.

Approximately 1 week before the date of the pretest interview, we emailed the participants a copy of the survey questionnaire and the frequently asked questions (FAQs) document along with instructions to complete the survey in advance of the interview and record the start and end times when they completed the survey (to provide an estimate of burden). On the day of the interview, the interviewer obtained the participant’s verbal consent and then used a debriefing guide to lead the participant through a discussion of their responses to the survey questions to identify any questions, response options, or terminology that was problematic. Of the six interviews that were scheduled (two in person and four by phone), three interviews were completed; one in person, one by phone, and one that was a combination of in person and phone.

Procedures for Informal Contacts with Industry Experts and Trade Associations

As part of the pretesting process, we made informal contacts with one industry expert and four trade associations. The industry expert previously worked for independent grocers but is currently employed by Walmart.

RTI identified the industry trade associations that provided feedback through a process of targeted Internet searches. In total, RTI identified over 60 trade associations that represent grocers, convenience stores, or retailers at the national or state level. Of these, RTI contacted 6 national trade associations and 21 state trade associations by email and/or phone to request their assistance in commenting on the survey instrument. Five associations agreed to informally review the SCANR Survey instrument and provide feedback:

California Grocers Association National Association of Convenience Stores Missouri Retailers Association

National Grocers Association Texas Retailers Association

Lessons Learned and Recommendations

We experienced some difficulties with recruiting and follow-through by the participants for the pretest interviews. RTI made nearly 60 calls to recruit store owners or managers using four different lists of stores from the sample frame. Table 1 summarizes the outcomes of these calls. Because we found that some stores were not eligible (e.g., no longer SNAP-authorized), we modified the computer-assisted telephone interviewing (CATI) script to attempt to ask the screening/eligibility questions if the participant declines to complete the survey so that when we calculate the response rate any cases determined to be ineligible are not counted as nonparticipants for the response rate calculation.

Table 1. Summary of Contacts to Retail Stores to Recruit for Pretest

Response	Number of Stores	Percentage of Stores
Ineligible	8	14
Unreachablea	32	55
Contacted and refused	12	21
Scheduled and completed interview	3	5
Scheduled but did not complete interview	3	5
Total	58	100%

^a Number no longer in service, wrong number listed in the sample frame, or no answer after three attempts.

Because many of the store owners are foreign nationals with English as their second language, we encountered language barriers, and many owners did not understand the purpose of the call or were unwilling to participate. We revised the recruiting letter, FAQ, and CATI script to help address some of these concerns. However, as previously shared with FNS, given the current political environment, foreign nationals may be reluctant to participate in the survey, thus potentially negatively affecting the response rate for the survey.

Another concern was reaching the store owner by phone. The script was amended to include mention of store manager if the owner is not available. When we could reach the owner or manager, they were often reluctant to participate. Many mentioned they are the only person in the store and could not commit to participating. This challenge suggests that our approach of offering multiple modes—mail, Web, and phone—will help maximize the response rate for the survey.

About 55% of the retail stores contacted were unreachable (i.e., number no longer in service, wrong number listed in the sample frame, or no answer after three attempts), which is a much higher percentage than we had assumed (15%) when we prepared the final study plan. This suggests that the contact data in the STARS database is not very current. For the full-scale study, we will contact participants by mail initially and then contact them by phone if a survey is not completed by mail or web. We will make up to 15 call attempts to cases contacted by phone and conduct a limited amount of tracing to obtain a working phone number. Because of the large percentage of unreachable cases, we increased the reserve sample for the survey in the final sample design.

Of the six interviews that were scheduled, three participants were not available when we tried to reach them by phone. We rescheduled and completed one of these interviews, but the remaining two were not rescheduled so an interview was not completed.

Detailed Findings and Recommended Changes to Instrument Based on Pretest Findings

Table 2 provides a question-by-question analysis of the pretest with a summary of the changes made to the instrument based on the pretest findings. Most of the changes dealt with simplifying the wording of the question or revising the terminology so that it was more familiar to retailers. No pretest interview participants found the survey difficult to complete.

Based on comments we received from several of the trade associations and discussions with our expert consultant, Art Burger, we revised Questions 12 through 17 that asked about the type of register system currently in use to follow the taxonomy shown in Figure 1. We will employ this same taxonomy for the cost analysis so that we consider the cost of complete system replacements versus modular upgrades to existing equipment.

Burden Estimates Based on the Pretest Findings

Table 3 provides a summary of the burden estimates we obtained from the pretest with retail stores. Two of the stores that participated in the pretest either did not complete the survey before the pretest interview or did not accurately record their time. Thus, most of the feedback on survey length is qualitative. However, all the feedback we received from pretest participants and trade associations suggests that the survey could be completed in less than 15 minutes and did not represent a significant burden. Thus, we will continue to use 15 minutes as the estimated burden for the OMB package.

Table 2. Question-by-Question Summary of Pretest Findings and Changes Made to Instrument

Original Question	Findings	Changes to Instrument
FAQ document	No concerns	Because it was difficult to recruit pretest participants, we revised the recruiting materials to address possible concerns about survey participation.
Recruitment letter	Add the store’s FNS number to the questionnaire to make it clear that the survey is for an individual store location.	Instead of adding the FNS number (from STARS database) to the questionnaire, we will include it on the recruitment letter to avoid having any information on the questionnaire itself that could be used to identify an individual survey participant.
Instructions	No concerns.	No changes necessary.

(continued)

Table 2. Question-by-Question Summary of Pretest Findings and Changes Made to Instrument (continued)

Original Question	Findings	Changes to Instrument
1. Is this store currently authorized to redeem SNAP/EBT benefits?  Yes  No  Complete Sections 1 and 2 then return the survey.	EBT can be used to dispense benefits from other government programs (e.g., WIC and TANF), so using “SNAP/EBT” may be confusing	Replaced “SNAP/EBT”’ with “SNAP” Replaced “redeem” with “accept”
4. How many cash registers/lanes are currently used by this store?  1  2  3  4  5  6 or more	Because previous questions ask about chains, clarify that remaining questions refer only to this store location. Some small stores may have up to 8 or more lanes. However, fewer than 1% of stores have more than 6 lanes. Therefore, we chose to use 6 lanes as the upper bound of our response options. The trade associations said that some stores refer to lanes as “points of sale” or “check stands.” However, none of the retailers we interviewed were confused by the “cash register/lane” terminology, so we did not make this change.	Revised Question 4. How many cash registers/lanes are currently used by this store?  1  2  3  4  5  6 or more
6. Is this store a WIC-authorized vendor? WIC refers to the Women, Infants, and Children Program.  Yes, have EBT system  Yes, do not have EBT system  No	EBT can be used to dispense benefits other than WIC. So simply asking if they have an EBT system may be confusing. Many retailers refer to WIC EBT as eWIC. Suggested we add “also” to make it clear we mean stores that are SNAP and WIC-authorized.	Revised Question 6. Is this store also a WIC-authorized vendor? WIC refers to the Women, Infants, and Children Program.  Yes, use paper vouchers  Yes, use Electronic Benefit Transfer system (eWIC)  No

(continued)

Table 2. Question-by-Question Summary of Pretest Findings and Changes Made to Instrument (continued)

Original Question	Findings	Changes to Instrument
7. How many unique barcode food products are sold in this store? Do not include products that are sold by weight. Remember that your best estimate is fine.  Fewer than 1,000 barcode food products  1,000 to 4,999 barcode food products  5,000 to 9,999 barcode food products  10,000 to 14,999 barcode food products  15,000 to 20,000 barcode food products  More than 20,000 barcode food products	Based on multiple interviews and responses from individual stores, we learned that small stores may have fewer than 1,000 items. Therefore, we made the response options more granular. To simplify the response options, we deleted “barcode food products.”	Revised Question 7. How many unique barcode food products are sold in this store? Do not include products that are sold by weight. Remember that your best estimate is fine.  Fewer than 100  100 to 499  500 to 999  1,000 to 2,999  3,000 to 4,999  5,000 to 9,999  10,000 to 14,999  15,000 to 20,000  More than 20,000
8. How many other unique food products are sold in this store that are sold by weight such as meat, fruit, or vegetables? These are sometimes called random-weight products or Price Look Up (PLU) products. Your best estimate is fine.  Fewer than 50 other food products  50 to 99 other food products  100 to 499 other food products  500 to 999 other food products  More than 1,000 other food products	The wording for this question was changed based on feedback that many convenience stores and other small retailers may not sell non-barcode items by weight. It was suggested to add hot deli items because these are also sold by weight, but because these items are typically not covered by SNAP, we did not make this change. Similar to Question 7, we made the response options more granular and simplified the response options.	Revised Question 8. How many other unique food products are sold in this store that do not have a barcode? These items are sometimes sold by weight and can include meat, fruit, vegetables and other items. Your best estimate is fine..  None  1 to 24  25 to 49  50 to 99  100 to 499  500 to 999  More than 1,000

(continued)

Table 2. Question-by-Question Summary of Pretest Findings and Changes Made to Instrument (continued)

Original Question	Findings	Changes to Instrument
9. How many full-time employees are currently employed at this store (including yourself, if appropriate)? By full time, we mean working at least 30 hours per week.  0  1–4  5–9  10–14  15–20  More than 20	Some stores we interviewed had only one employee, so it may be useful to make the response options more granular. The industry standard for full time is 35 hours per week.	Revised Question 9. How many full-time employees are currently employed at this store (including yourself, if appropriate)? By full time, we mean working at least 35 hours per week.  0  1 or 2  3 or 4  5–9  10–14  15–20  More than 20
10. How many part-time employees are currently employed at this store (including yourself, if appropriate)? By part time, we mean working fewer than 30 hours per week.  0  1–4  5–9  10–14  15–20	Same comments as above	Revised Question 10. How many part-time employees are currently employed at this store (including yourself, if appropriate)? By part time, we mean working fewer than 35 hours per week.  0  1 or 2  3 or 4  5–9  10–14  15–20  More than 20
11. How many of your full- or part-time employees are primarily responsible for checking out customers?  1–4  5–9  10–14  15–20  More than 20	Same comments as above	Revised Question 11. How many of your full- or part-time employees are primarily responsible for checking out customers?  0  1 or 2  3 or 4  5–9  10–14  15–20  More than 20

Table 2. Question-by-Question Summary of Pretest Findings and Changes Made to Instrument (continued)

Original Question	Findings	Changes to Instrument
12. What type(s) of payment terminal(s) does this store use? By payment terminal, we mean the device used to take the customer’s payment (also known as a point-of-sale terminal or credit card terminal). [Select all that apply]  Installed (e.g., PC/Mac)  Tablet  Mobile (smartphone based)  Other (Please specify): ___________________	The use of the term “installed” is confusing; the more commonly used term is “integrated.” Terminology was confusing; add examples (e.g., NCR, IBM). Based on these comments, additional findings from the follow-interview pretests, and discussions with our expert consultant, we modified our approach for collecting information on the retailer’s existing system to mirror the taxonomy shown in Figure 1. We revised Question 12 to capture information on whether the store’s register system is integrated with the EBT payment terminal. For our analysis, we do not need information on the type of hardware used.	Revised Question 12. Is your store’s front-end register system integrated with the EBT payment terminal?  Yes  No, we must enter SNAP transactions in both the register and payment system.
15. Does your store’s register system scan UPC codes on products during checkout?  Yes, currently operational  Yes, in the process of purchasing/installing  No  Go to Question 18	Some stores refer to UPC codes as SKUs. Questions 7 and 8 use the term “barcodes”; the questions should be consistent.	Revised Question 15. Does your store’s register system scan barcodes on products during checkout?  Yes, currently operational  Yes, in the process of purchasing/installing  No  Go to Question 18

(continued)

Table 2. Question-by-Question Summary of Pretest Findings and Changes Made to Instrument (continued)

Original Question	Findings	Changes to Instrument
16. Does your store’s register system have indicators/flags on food products that are eligible for purchase with SNAP/EBT?  Yes, currently operational  Yes, in the process of purchasing/installing  No  Go to Question 18	See previous comment on Question 1 about SNAP and EBT. Revised question wording because some respondents may not know what we mean by indicator/flag.	Revised Question 16. Does your store’s register system identify products that are eligible and not eligible for purchase with SNAP benefits (for example, by using a flag or other indicator)?  Yes, currently operational  You have completed the survey  Yes, in the process of purchasing/installing  You have completed the survey  No  Answer Questions 17 and 18
17. Does your store’s register system prevent the authorization of benefits for food products that are not eligible to be purchased with SNAP/EBT (this is referred to as an integrated system)?  Yes, currently operational  Go to Section 4  Yes, in the process of purchasing/installing  Go to Section 4  No	See previous comment on Question 1 about SNAP and EBT.	Deleted this question because Question 12 was revised to capture information on whether the register system is integrated.

(continued)

Table 2. Question-by-Question Summary of Pretest Findings and Changes Made to Instrument (continued)

Original Question	Findings	Changes to Instrument
18. The 2014 Farm Bill will require all SNAP-authorized retailers to (1) use scanners at checkout to accept SNAP/EBT benefits and (2) include in their store’s register system indicators/flags on SNAP/EBT-eligible products. Knowing what you know today, how likely are you to upgrade or purchase technology within the next 2 years so that you meet this requirement and remain a SNAP/EBT-authorized retailer?  Very unlikely  Somewhat unlikely  Neither unlikely nor likely  Somewhat likely  Very likely	Revised question so it is clear the requirement is not in place yet. Otherwise, participants may think they are out of compliance and be reluctant to return the survey. Additionally, the previous wording seemed to imply that the store owner would have to make equipment purchases within the next 2 years before knowing what the USDA’s equipment requirements would be. We revised to address these concerns and to remind the retailer that they may need incur additional and ongoing costs Same comment as on Question 1 regarding SNAP/EBT.	Revised Question (now Question 17) 17. There is a new law that will require all SNAP-authorized retailers to use scanners at checkout to accept SNAP benefits. In the future, your store may need to upgrade or purchase and maintain new equipment to comply with this law. How likely are you to do this so you can remain a SNAP-authorized retailer?  Very unlikely  Somewhat unlikely  Neither unlikely nor likely  Somewhat likely  Very likely
19. In the table below, indicate how important each of the factors would be to your decision on whether to upgrade or purchase scanning technology that meets the new Farm Bill requirement.	Add “time to maintain product database” as a barrier because this can be very time consuming and possibly a barrier to adoption.	Added the following factor to the table (now Question 18): “Time to maintain product database” Replaced “new Farm Bill requirement” with “new requirement”

(continued)

Table 2. Question-by-Question Summary of Pretest Findings and Changes Made to Instrument (continued)

Original Question	Findings	Changes to Instrument
Please provide your contact information below. This will only be used if your store is selected to take part in a follow-up interview, either by phone or in person.	It was suggested to add text to this question to make it clear that all information participants provide will be held confidential. Based on this concern and the difficulties we had recruiting retailers for the pretest, we have decided to drop this question from the survey because contact information will be available in the STARS database. We are concerned that most participants would say they do not want to be contacted; then we would be unable to contact them again. The FAQ indicates that they may be contacted in the future, which is required as part of the informed consent process.	Deleted question.

Table 3. Pretest Participants for SCANR Questionnaire and Estimated Time and Opinion Regarding Survey Length

Participant	Time to Complete	Opinion on Survey Length
Convenience store Wilkesboro, NC (rural)	<10 min	The survey was not difficult to complete and the length seemed about right.
Indian grocery store Cary, NC (urban)	Not available	Although the retailer did not complete the survey before the call, when asked about the length, he said he could have completed it quickly.
Convenience store Raleigh, NC (urban)	Not available	The retailer did not keep an accurate measure of the time it took to complete but did not feel completing the survey was excessively burdensome.

PRETEST OF follow-up interviews with retailers

Five pretests were conducted of the follow-up interview guide. Valuable information was learned from each of these pretests that will be useful for the full-scale study. The following sections begin by discussing the description of the pretest participants, the procedures followed for the pretests, and the lessons learned and recommendations for the full-scale study. We then provide information on the burden estimates based on the pretest findings. We conclude with the recommended changes to the follow-up interview guide.

Pretest Participants

To pretest the follow-up interview instruments, a project team member located in Washington State was given a list of eligible stores. The list included the name and owner contact information for stores within three counties in Washington: Thurston, Pierce, and Kitsap. An additional store was selected in New Mexico where one of the project consultants resides. The number of desired pretest participants was at least four, including at least one that did not have a scanning system in place that would meet the new federal requirements and at least one that had a scanning system that meets the requirements.

Because the stores in Washington were in a large geographic area and there was no information available regarding whether scanning systems were in place, stores within a close geographic proximity were screened through initial visits to identify the types of business, whether the stores had scanning systems, and whether the store owner/operator speaks English. A total of 25 stores were screened, and 12 stores were good candidates for the pretest. After identifying eligible stores, the project team member made a second visit to discuss the purpose of the study and the pretest and gain cooperation from the owner/operator. Six stores in Washington agreed to participate and times were scheduled for interviews. When the interviews were scheduled, the project team member explained to the owners that only four pretest interviews were necessary and their interview would be cancelled if enough were done before the time scheduled for their interview. This approach provided alternates for store owner/operators who were not available at the scheduled time or who changed their mind and declined to participate.

The project consultant in New Mexico visited the store identified in that state to explain the purpose of the study and pretest and ask the store owner to participate. The owner was willing to take part in the pretest and a time was scheduled for the pretest interview.

Of the seven stores that were recruited for the pretest:

• Three were small full-service markets, each with one checkout stand and with scanning equipment.

• Two were small stores with a gas station that had scanning equipment.

• One was a store specializing in Asian foods and had scanning equipment.

• One was a small “country” store in a rural area that uses a computer (no scanning equipment) for checkout.

It is important to note that some of the Washington stores that would have otherwise been included did not want to participate in the pretest if participating was not a requirement associated with SNAP. These store owners were primarily of Hispanic, North African, or Middle Eastern decent. They indicated or implied that they did not want to provide the government with any information unless they were compelled to under federal rules. Although not specifically stated, the current political environment may have been a factor in their unwillingness to participate. Also, a small number of stores were excluded because of the owner’s limited English-speaking abilities. These stores were owned by individuals who spoke only Asian languages and would have required interpreters to complete the interviews.

Procedures

At the time of the appointment, the project team member arrived at the store and asked for the owner, explaining that the project team member had scheduled an interview with him/her. Three of the Washington store owners and the New Mexico store owner were available at the scheduled time, and interviews were conducted with them. At the other Washington stores, the owner was not there but was expected to return later in the day. The team member returned to one of these stores multiple times until one additional interview could be conducted. In one store where the owner agreed to be interviewed and was at the store, the owner’s limited English skills required that he call his daughter (who also worked at the store) to help interpret the questions and answers.

A total of five pretest interviews (four in Washington and one in New Mexico) were conducted. Table 4 provides the store descriptions and time spent on the pretest interviews.

Table 4. Pretest Participants’ Time and Opinion Regarding Interview Length

Participant	Time to Complete, Minutesa	Opinion on Length
Small full-service grocery in Purdy, WA	50	Time required to complete interview heavily influenced by interruptions. Owner thought length of interview was acceptable.
Asian food market in Tacoma, WA	60	Owner found it difficult to understand some of the questions because of language barrier. Owner’s daughter assisted as interpreter. Owner found the length of time for the interview acceptable.
Small store with gas station in Gig Harbor, WA	35	Took less time because it was conducted early morning and store was not busy. Owner found the length of time to complete the interview acceptable.
Small rural market in Gig Harbor, WA	45	Store does not have scanning equipment and will drop SNAP if required to do so. Owner fine with the amount of time survey took.
Small store with gas station in Santa Fe, NM	60	Store has price look-up scanning equipment but does not identify SNAP-eligible foods. System is focused on tracking gas sales and meeting tax requirements. Owner found the time to complete the interview acceptable.

^a Time to complete the interviews included multiple interruptions; time spent on questions estimated to average 25 minutes per interview.

Both project staff who conducted the pretest found that they had to be more conversational with the participants; consequently, the questions did not get asked in the order on the guide. The interviews were frequently interrupted because the participants needed to address employee or customer questions or because the participant was both the owner and the cashier. The project staff were still able to ask most of the questions with this approach but not in the order in the structured guide. In addition, for the pretests, the project staff ended up using a combination of questions from both the guides because it was not clear when the interview began if the equipment the store had could identify SNAP-eligible purchases.

Lessons Learned and Recommendations

Several lessons were learned from the pretest and will be useful in scheduling and conducting the follow-up interviews with retailers.

1. Because recruiting stores is likely to be challenging, flexibility in the recruitment and interviewing approach will be necessary.

Even though the number of stores recruited for the pretest was small and store owners/operators were not familiar with the study, in-person recruitment and interviews are likely to be more successful than phone calls. Although we will recruit stores that completed the retailer survey and will, therefore, be familiar with the study, we anticipate needing to use a flexible approach to recruit and conduct the interviews. To increase the opportunities for recruiting and conducting the interviews, we will select stores in clustered geographic areas where trained interviewers are located (e.g., within 60 miles of an interviewer). After the recruitment letters are sent to the store owners, interviewers will make at least three attempts to call owners/operators to schedule the interview.

For stores with scanning equipment, interviewers will schedule an onsite interview with owners reached by phone. If interviewers are unable to reach the store owner by phone, interviewers will go to the store to attempt to meet with the owner and request cooperation. If owners are unable to participate in an interview at that time but are willing to schedule an interview at a later date, interviewers will return. If owners are willing to participate but ask the interviewer to wait for a time when they are less busy, interviewers will remain onsite until the interview can be completed. This approach will accommodate the reality that store owners may not be able to commit to an exact time for an interview. It will also address language barriers in situations where owners may ask interviewers to wait until an employee can help interpret.

For stores that do not have scanning equipment, interviewers will attempt to conduct the interviews by phone. If after multiple attempts, they are unable to reach the owner or the owner is unable to participate in a phone interview (e.g., language barrier), interviewers will attempt to do these interviews in person if the stores are located within the geographic areas where interviewers are located. The interviewers will use the same approach as described above for the stores with scanning equipment.

Based on the pretest, this flexible approach will facilitate conducting 50 interviews. The pretest experience indicates conducting phone interviews with store owners/operators will be challenging, in part because they need to be accessible to customers and employees and may be interrupted frequently during the interview. Language barriers will also make phone interviews less viable than onsite interviews for some stores.

2. Although the time necessary to answer the questions is reasonable, the constant interruptions that occurred with store owners extend the time needed to complete the interview.

One of the key issues faced in the pretest is that store owners are frequently interrupted by customers and employees. In some cases, the store owner was the only one there to conduct the store’s business (e.g., work at the cash register). In other cases, there were questions from clerks or from customers that only the owner can answer. Thus, although asking and answering the questions would take about 20 to 25 minutes to complete in a nondisruptive environment, interviewers will need to plan to be at the store for up to an hour to complete the interview given the likelihood of interruptions. Additional time may be required if the participant is busy at the time of the scheduled interview.

3. Language barriers may be significant.

A number of the store owners initially screened for selection in the pretest had difficulty understanding English. Language barriers will make phone interviews more challenging than in-person interviews and may require that some of the interviews intended to be done by phone will need to occur at the store. In some cases, another person working in the store may be fluent in English and available to assist.

4. Highly structured interviews are not an effective way to collect data from this target group. Rather, conversational interviews with the owners are likely to achieve the best results.

During the pretest interviews, it was clear that trying to follow a structured interview guide question by question was not the most effective way to collect data, especially given the disruptions that occurred during the interview. Conversations with the store owner that incorporate the questions into the discussion worked best. As noted, both project staff who conducted the pretest had to be more conversational with participants and did not ask the questions in the order on the guide. Interviewers will need to be trained to handle these interviews in a less formal manner than a typical key informant interview. These interviews will need to be conducted in a semistructured, case study approach.

Through the pretest, we identified three categories of store systems: (1) cash register and scanning system that is linked to the inventory system so that SNAP-eligible purchases are identified at checkout; (2) cash register and scanning system that does price look up and is primarily used to address tax issues related to tobacco, liquor, or gasoline sales but does not identify SNAP-eligible purchases; and (3) cash register system that does not scan purchases or identify SNAP eligibility of an item. For the second group, questions need to be directed toward modifications to an existing system rather than acquiring a new system.

Because the SCANR Survey will collect information that will help identify the type of system in the store, the type of system should be less of an issue for the stores selected for the interviews. However, if the survey participant is not knowledgeable or does not answer the questions correctly, interviewers will need to be prepared to modify the questions during the interview. We will need to train data collectors on how to potentially ask questions from both interview guides if necessary.

5. Cost questions will be difficult for participants to answer.

Most pretest participants had very little information regarding actual or anticipated costs. One store had recently upgraded its equipment and provided a copy of a receipt (from a gasoline company that installed the system, primarily to record gas sales and tax), and another shared that the system they selected was one promoted by the Korean Grocer Association, and maintenance of their equipment is handled via a monthly fee to the association. One other participant indicated that the system was installed several years ago and they did not have access to the cost information. Another noted that the system was in the store when he purchased the store, so he had no idea what the cost had been. Because small stores have a history of high owner turnover, this may be the case in many stores included in the study.

Also, for the stores with price look-up but no ability to identify specific SNAP-eligible foods at point of sale, the two owners said that they have not assessed what it would cost to upgrade, because their SNAP sales are not enough to justify the cost of making changes. They feel that their systems “work fine now” and upgrades are not worth the expense. These stores indicated they would drop out of SNAP if they were forced to upgrade.

Store owners, for the most part, did not track how much time was put into installation and training on their systems. Some were able, however, to estimate how much time it takes to train a new employee on using the cash register system. One owner noted that it takes about an hour to train a new employee but trying to determining the dollar value for that time was not reasonable.

Our conclusion from the pretest is that the interviewers will be able to collect some valuable qualitative information to supplement the survey responses, but the amount of information and breadth of detail will vary greatly among the interview participants. The responses may be more reflective of case study descriptions than structured interview data. The open-ended, “big picture” questions in the interview guides should be sufficient to address the research questions identified for the follow-up interviews. However, some of the detailed probes in the interview guide are unlikely to be asked of many participants.

Burden Estimates

Table 4 summarizes the pretest participants, the time to complete the pretest interview, and opinions regarding interview length. Although the time in the store ranged from 35 minutes to up to an hour due to interruptions from normal store activities, the actual amount of time required to complete the interview was about 25 minutes. We had estimated a 30-minute burden, so we will use continue to use a 30-minute burden in the OMB package.

Recommended Changes to Interview Guide

Tables 5 and 6 provide a summary of the pretest findings and suggested changes to the instruments. Separate tables are provided for the interview guide for stores with scanning technologies that meet the Federal requirement (Table 5) and stores without scanning technologies that meet the Federal requirement (Table 6).

Table 5. Interview Guide for Follow-Up Interviews: Stores with Scanning Technologies that Meet the Federal Requirement

Question	Findings	Changes to Instrument
Please show me the scanning system you are using and describe how it works.	All owners could show and demonstrate their systems.	Probe question implies that current owner may have considered other scanning technologies when, in fact, the scanning technology came with the store when purchased or was provided by a supplier (e.g., gasoline). Also, the owner may not have installed the scanning technology but may have updated it. Changed probe questions b, c, and d to reflect possibility that the scanning technology came with the store or may have been updated by the current owner. Added screening probe, changed verb tense where appropriate, and added a probe about if they now wished they had looked at other scanning technologies.
Thinking back to when the system was installed, I’d like to know what changes you made in the store to set it up and what you experienced during the installation.	Mixed results with this question. Some owners were not the owner when the scanning technology was installed. Others could not remember much about the installation. One could describe the process well and could explain disruptions and problems.	Refined the question to reflect that store owner may not have been the one who purchased the scanning technology but may have updated it. The probe questions can then apply to system updates. Added phrase “thinking back to when the scanning technology was installed, or if you have updated the scanning technology, …” and added update to the probe questions.
I’d like to learn about the costs of installing and operating the scanning system. It is okay to provide estimates or a range if you don’t know the exact amount. Just tell me it’s an estimate.	This question will be hard for many to answer, but we can possibly get information about costs of updates or ongoing servicing and maintenance. Two participants were able to give some cost information. Also, none of the participants leased the scanning technology, so we do not know how they will respond to the questions pertaining to leased scanning technologies.	No need to omit or significantly revise; however, many participants may be unable to give specific information about entire scanning technology from installation to maintenance. Modified probes because some owners who updated their scanning technologies may be able to share cost information.

(continued)

Table 5. Interview Guide for Follow-Up Interviews: Stores with Scanning Technologies that Meet the Federal Requirement (continued)

Question	Findings	Changes to Instrument
I’d like to hear about any technical issues or problems you have when using the system either at the time it was installed or during the time you’ve been using it.	Owners were able to talk about specific issues related to upgrades or system problems.	No changes needed other than adding option to ask about scanning technology updates, if applicable.
How well does this scanning system meet your needs? What makes you feel it [does/does not] work well for your business?	Owners were able to answer this question.	No changes needed.
Would you recommend your scanning system to other stores that do not have scanning systems but might be looking for one? If so, why? If not, why?	Owners were able to answer this question.	No changes needed.

Table 6. Interview Guide for Follow-Up Interviews: Stores without Scanning Technologies that Meet the Federal Requirement

Question	Findings	Changes to Instrument
First, I would like to hear about the system you currently use for transacting sales, in other words, your cash register or point-of-sale system. Please describe the system you use.	The one store without scanning technology could show and demonstrate their cash register system.	Because some stores have scanning technologies but their register system does not identify SNAP-eligible items, this question was expanded to ask for a description of the capability to scan items. Also, added probes to ask if the participant selected the scanning technology and, if so, why they chose it.
As you may know, for your store to continue to be authorized for SNAP/EBT in the future, you will need to use scanning technologies to identify SNAP/EBT-eligible products. Are you currently considering implementing a scanning system?	Store owners knew nothing about requirement.	At the start of Question 2, removed “as you may know” and replaced with “as I explained when we started the interview …” This lead-in provides a clearer statement explaining this requirement is new. Also, revised questions to apply to system updates in addition to new systems.

(continued)

Table 6. Interview Guide for Follow-Up Interviews: Stores without Scanning Technologies that Meet the Federal Requirement (continued)

Question	Findings	Changes to Instrument
Where do you get information about store technology, including scanning systems?	Store owners did not know where to get information, but they are not considering purchasing new scanning technology now. One said he would talk to other store owners he knows if he decides to change his store technology.	Many of the probes will not be asked/answered if the owner has not been looking for a new scanning technology. Modified probes to apply to system updates, as well as new systems, and to differentiate between stores that have scanning capability and those that do not have it.
Next, I have some questions about potential costs you would incur when implementing a scanning system.	Owners could not answer this question because they have not considered a new system or an update.	Revised questions to apply to scanning technology updates, as well as new technologies, and to differentiate between stores that have scanning capability and those that do not have it. Also, because some owners may have considered a new system but rejected it because of many of the factors asked about in the probes, added this possibility to the interviewer instructions.
Overall, what are the most important factors or concerns you have related to installing a scanning system? Why are those most important?	Owners could not answer this question because they have not considered a new system or an update. Most owners are concerned about having systems that meet state and federal tax requirements for tobacco, alcohol, and gasoline sales. For these stores, SNAP is a small part of store sales.	Add scanning technology update to the question.