24_W. Pretest Methods and Summary of Findings

W. Pretest Methods and Summary of Findings

Pre-test Methods and Summary of Findings

Pre-tests were conducted with the three eHIP States, Colorado, Washington, and Louisiana. Westat sent each State the revised cost templates on March 6. We asked the States to review the templates in advance of a two-hour meeting where the templates would be discussed. We also asked each State to send their availability for this meeting as well as all individuals they wished to be invited.

We aggregate all feedback from the States.

1. Pre-test Feedback and Suggested Response

Administrative Cost Workbook
Tab	Feedback	Suggested Response
Overall Workbook Feedback	State concerned about the granular nature of the reporting	Emphasize that estimates can be provided where data are not expressly tracked Explain that items that cannot be estimated can be discussed during State interviews (during data collection)
Quarterly Expenses	State said that contractors may not be charging the grant for all personnel time	Make sure to include questions about personnel time not charged to State in contractor interviews
	State said that farmers may be a unique contractor expense, with tasks different than those for other retailers	Consider adding a category for farmers under contractor expenses, or add example of farmers under “other” contractor expenses
	State may be providing additional assistance to retailers for startup costs	Provide additional guidance in instructions about how to categorize different types of retailer assistance
	State was unsure what types of contractor costs should be included	Clarify that only contractor costs for eHIP work should be included
Activities	State said that farmer recruiting and startup support could be another activity; because different than other retailer activity, should be separate activity	Consider adding activity for farmer recruiting and startup support; if this is added, retailer recruiting and startup should be changed to non-farmer retailer recruiting and startup (or something similar)
	State was unsure of where time spent on contracts should be placed	Clarify that contract activities for retailer contracts should go under retailer recruiting and startup support; contract activities for other contracts should go under project management
	State was unsure whether “fraud” included SNAP fraud	Clarify that “fraud” is only eHIP fraud, not SNAP fraud
Personnel Hours by Activity	State asked whether they should estimate the time spent by local offices handing out brochures or mentioning eHIP as part of eHIP promotion	Clarify in instructions that promotion includes development of promotion materials, planning promotional campaigns, or similar, and does not include minimal time spent by local offices in handing out brochures or the like
	State asked whether all time on eHIP should be tracked, including minimal time answering emails or similar	Clarify that time estimates should be as precise as possible, but we are not expecting every minute to be accounted for Clarify further that the focus should be on significant staff and significant activities; staff who attend a meeting where eHIP is incidentally mentioned, for instance, would not be included
	State asked whether the Activity could be made into a drop-down list, to reduce the amount of going back and forth between tabs	Make Activity a drop-down list
Salaries	State asked whether loaded rates could be included instead of unloaded rates	Emphasize in instructions that either unloaded rates + fringe rates should be included OR loaded rates (this is in the column header but should be emphasized) In column headers, further differentiate between loaded and unloaded rates
	State said some staff may not be comfortable divulging salary	Clarify that salaries can be provided as rate for the specific staff or as the midpoint in the range for the staff category
Non-Personnel Costs by Activity	State asked how to allocate costs if they are related to multiple activities (e.g., if payments for retailers are for both retailer recruitment and retailer support)	Clarify in instructions that costs can be allocated to activities by estimate, if precise values cannot be given Note in interview protocols that additional probing may be required to estimate the proportion of costs for each activity
	State asked about where to include travel expenses	Clarify that travel expenses can be included under “Other”; alternatively, could include a separate line item for travel If a separate line item for travel is included, this line item should also appear in the Quarterly Expenses tab

Incentive Report
Tab	Feedback	Suggested Response
Overall Workbook Feedback	State found instructions a bit lacking across these tabs	Add additional detail and examples in the instructions
Summary	State found the instructions unclear; wanted more examples and more clarity on what is optional	Provide additional instructions, with special attention to what sections are optional and what sections are not, as well as more instructions on the meaning of the data sought Use color to distinguish optional sections
	State found it unclear if additional information was needed for items where said (if this is gross before refunds…”)	Clarify in instructions that no additional information needs to be provided if State provides net figures
	State wanted to clarify whether number of transactions earning incentives excluded refunds	Clarify that we are seeking the number of transactions earning incentives (excluding refunds)
	State was unclear about how they should calculate averages	Make lines asking for averages optional, or clarify that if the EBT report provided to the State does not provide averages, the State should not try to calculate it on its own
	State mentioned that some of their data will be for cardholder, rather than households	Ask States to mark in the Notes column if any item is provided for the cardholder instead of the household
Retailer list	State asked about how often they would need to submit the retailer list	Consider either clarifying that this list needs to be submitted once at the beginning of data collection and only subsequently if outlets are added or drop off OR make the retailer list a separate file with similar instructions
	State asked about identifying implementation method for farmers’ markets	Consider adding a separate implementation method for farmers’ markets (perhaps “farmers’ market solution”)
	State asked about retailers using apps as the implementation method	Add “App/automated” or something similar as an option for implementation method
	State asked about the store type column	Add link in instructions to USDA information on store type
	State asked whether they should provide the FNS ID number for stores or the store’s own ID number	Clarify that the State can provide either the FNS ID or the store ID—as long as they are consistent between the Retailer list tab and the Retailer-level incentives tab
	State asked what they should answer if they were not sure about whether a store was offering a non-eHIP incentive	Add a “don’t know” option for all of the non-eHIP incentive columns