Agricultural Labor Test - Detailed Testing Plan

Agricultural Labor Survey, Testing Plan

At the recommendation of OMB, NASS plans to pilot test the implementation of the Standard Occupational Classification Codes (SOC) during the October 2013 data collection cycle. Our plans will involve a two step process discussed below.

Qualitative Research Plan

Goals of the qualitative research:

1.) Determine whether agricultural operators understand the categories.

2.) Determine whether these operators can accurately report the labor on their operation using these proposed categories.

The qualitative research will be conducted in July 2013 and will consist of cognitive interviews with a sample of 25-30 operations from the April Survey. This will provide a sample of respondents who have filled out the current version of the questionnaire. As respondents fill out Section 1, they will be asked to think aloud and describe what they are thinking and how they came up with their answers. RDD will examine the affect of using the new version with the additional field and livestock labor categories, as well as the respondent’s ability to fill out the information for the current quarter and the previous quarter.

To contain costs, staff with survey methods training in the field offices will be tasked to complete some of the cognitive interviews in select operations in GA, MO, VA, MD, and WA.

The cognitive interviewers will ask the respondent to think aloud as they answer the questions and then probe any areas where the respondent struggles. This should be the majority of the discussion about the categories and the issues that operators have using the categories to answer the questions in Section 1.

General cognitive interviewing probes will be used such as ‘how did you arrive at that answer?’, ‘what made that question/category confusing for you?’ and ‘how could that question/category be improved?’

Possible additional questions might include:

• What types of work do you hire others to do on your operation? (this could be a long list)

  • Are all of those types of work covered by these categories?

• Is it possible for you to assign a category to each of your workers? If not, what would help you choose one of these categories to describe your occupation?

• Do most paid workers on your operation do more than one type of work?

• How would you record these workers on the questionnaire?

• Please look at the example given for each category. Are there other examples that you would include for any of the categories?

• Did you have any difficulty using the current/new categories? Which was easier? Why?

After reviewing the results of the cognitive testing, the Agricultural Labor Survey team will update the questionnaire for testing in October 2013.

Quantitative Research Plan

During the October data collection, NASS will use the current sample of 12,000 operators and ask them to report using the current OMB approved questionnaire. A supplemental questionnaire using the re-formatted table that incorporates the SOC codes will be included at the end of current questionnaire. Data from the current questionnaire will be used to produce estimates for the October 2013 official estimates. Data from the questionnaire supplement will be used to create a separate estimate that will be compared to the estimate created from the current questionnaire. The re-formatted table will be based on the results of the cognitive testing conducted in July of 2013. Interviews during the October data collection will be conducted by mail, phone, and field follow up.  For face-to-face interviews, interviewers will use a paper supplement to ask the questions of the respondent using the new categories. All CATI interviews will include the new questions at the end of the interview. All operations that complete the interview by mail will be called back to answer the supplemental questions.

The supplemental form will collect data for both July and October reference weeks (as does the operational survey) to assess differences in recall bias on the test questionnaire tables compared to the operational program.

For both July and October reference weeks, data from each questionnaire table version, along with the corresponding sampling and nonresponse-adjustment weights, will be processed through the operational Ag Labor edit and summary programs.  The summary program will generate aggregate totals along with variance calculations at the national and Ag Labor Regional level (Corn belt 1, Corn belt 2, etc.) for variables of interest to this study, including a calculated sum of parts equal to the Field, Livestock, Supervisor, and Other categories as asked in the operational survey.  The variables of interest are the number of hired workers, the total hours worked, the average hours per worker, the total wages paid, and the average wage rate paid in dollars per hour by type of worker.

The data for this study will be analyzed by comparing individual responses and summary totals between the current operational and test questionnaire table versions.  The dataset will consist of the reported values and the corresponding weights for each questionnaire table version.  The expanded values will be calculated by taking the product of the reported values and corresponding weights.  T-tests will be performed on both record level data and expanded values at the national and Ag Labor Regional levels to determine whether the means of the reported data were equal for each of the variables in Field, Livestock, Supervisor, and Other categories.

After the data is released in November 2013, the NASS survey team will review the data collected and determine next steps for implementation of the SOC codes into our survey instrument and publication.