NASS Comments on MOLS

OMB Docket from AMS: Customer Satisfaction Survey for the Marketing Order Online System (MOLS)

The materials transmitted appear to be incomplete; they also show some inconsistencies. In part A of the documentation there is mention of a five-point Likert scale, but in the tentative questionnaires included, most Likert-modeling questions reflect a ten-point scale; OMB should be given a copy of the questionnaire in nearly final form, probably without “Survey Monkey.” It should be explicitly stated (in Part B: 2 of the documentation, which was misinterpreted) that the survey proposed is actually a census of MOLS users: all online MOLS users (requestors of exemptions from import inspection) are being asked to complete the survey (a 100% sample). Results of the survey are not intended for publication, but merely for intra-agency use, to identify ways to improve MOLS and make it more accessible to users.

Part B contains a critical omission: although a nonresponse follow-up is mentioned, no other procedures were cited for dealing with nonresponse; no nonresponse weighting methodology was mentioned and there is no substantive discussion of the proposed nonresponse follow-up. A conventional weighting solution would involve devising a simple form of nonresponse stratification for MOLS users, based on type or size of operation; response could then be treated as a kind of sampling process, with respondents weighted by stratum by the inverse of their stratum-level response rate. In any case, with only a 20% response rate anticipated, the significance of the Likert medians or means would remain unclear; the survey data might not be informative. Without intensive follow-up, the survey responses may incur unknown biases; intense dissatisfaction of a vocal minority could cloud the results, as could frenetic applause from a highly supportive subgroup.