Summary of Ag Labor Cognitive Interviews

Summary of Ag Labor Cognitive Interviews

Conducted in July and August 2013

Finalized by Kathy Ott, NASS/RDD, September 4, 2013

1. BACKGROUND

At the recommendation of OMB, NASS will conduct quantitative testing of the use of the Standard Occupational Classification (SOC) Codes on the Agricultural Labor Survey during the October 2013 data collection cycle. In preparation for that test, qualitative work in the form of cognitive interviews was conducted in July and August 2013.

The Agricultural Labor Survey is conducted twice a year in April and October. Data are predominantly collected via mail, with CATI follow-up via the Data Collection Center (DCC), with some in-person enumeration.

Section 1 of the Ag Labor Survey asks about the paid workers on the operation. Respondents are asked to categorize the workers by the type of work these workers were primarily hired to do during a particular reference week.

Currently, there are four categories:

1. Field Workers

2. Livestock Workers

3. Supervisor/Managers

4. Other Workers

NASS is now required to test and possibly use a more detailed breakdown of these categories called the Standard Occupation Codes (SOC). Categories developed and used for this phase of testing are:

1. Field Workers:

1. Agricultural Equipment Operators – Crop, Nursery and Greenhouse

2. Farmworkers – Crop, Nursery and Greenhouse

3. Graders and Sorters – Crop, Nursery and Greenhouse Products

4. Hand Packers and Packagers - Crop, Nursery and Greenhouse Products

5. Agricultural Workers, All Other - Crop, Nursery and Greenhouse

2. Livestock Workers:

   1. Farmworkers – Farm, Ranch, and Aquacultural Animals

   2. Graders and Sorters - Farm, Ranch, and Aquacultural Animal Products

   3. Hand Packers and Packagers - Farm, Ranch, and Aquacultural Animal Products

   4. Agricultural Workers, All Other - Farm, Ranch, and Aquacultural Animals

3. Supervisors

   1. Farmers, Ranchers and other Agricultural Managers

   2. First-Line Supervisors of Farm Workers
      
      

4. Other Workers

   1. Agricultural Inspectors

   2. Animal Breeders

   3. Pesticide Handlers and Sprayers

   4. Any Other Worker Not Listed Above

The cognitive testing focused on just the categories used for the survey, but other issues were identified if they came up during the testing.

Goals of the cognitive interviews:

1.) Determine whether agricultural operators understand the categories.

2.) Determine whether these operators can accurately report the hired labor on their operation using these proposed categories.

All of the questionnaire pages that were used in the test are included in the Appendices.

2. COGNITIVE INTERVIEW SAMPLE SELECTION

With limited resources and without knowing the extent of problems or issues that might come up with the new categories, we hoped to complete 25-30 total cognitive interviews for this project, across several states.

Because of resource constraints (both budget and personnel), sample for the cognitive interviews was selected in states where we have staff with some training or experience in conducting cognitive interview recruiting and interviews. The states selected were MO, GA, WA, MD, and VA.

Because the focus of the study is categorizing agricultural labor and we wanted respondents who might have some knowledge of our current categories, we limited the sample to respondents who participated in the April 2013 Ag Labor survey and who reported at least one hired laborer. A total of 24 operations were sampled for each participating state, with the intention of getting a total of 5-6 participants per state.

Each cognitive interviewer was responsible for recruiting participants in their state(s). To contain travel costs and staff time, in general, interviewers attempted to contact operations that were geographically convenient to their home or office first.

Twenty four total cognitive interviews were conducted for the Ag Labor survey in July and August 2013.

3. COGNITIVE INTERVIEW SUMMARY

Two rounds of cognitive interviews were completed using the protocol shown in Attachment E. Seventeen of those interviews were completed in the first round. In this round, we tested the original categories and the first version of the revised categories. Appendix A contains the current questionnaire pages that use the original categories from the April 2013 Ag Labor Survey. Appendix B contains the first version of the revised categories. Following Round 1, we made changes to the revised categories. This second version was then tested in Round 2, with 7 respondents. The second version of the category table is shown in Appendix C. Table 1 below summarizes the states where the 24 interviews were done for each round.

Table 1: States included in the cognitive interviews

State	Number of interviews using first version (Round 1)	Number of interviews using second version (Round 2)
GA		6
MD	6
MN*	1
MO	5
VA	3	1
WA	2
Total	17	7

*we added one operator from MN who served as a good initial case for a new cognitive interviewer

Tables 2 and 3 in Appendix D give a brief description of the operations in each round of testing.

1. Round 1 Summary

As mentioned earlier, the first revised version of the categories was used for the first 17 cognitive interviews. Half of the respondents were asked to fill out the questionnaire using the current version of the categories first and half were asked to use the revised version first. The issues discovered in those 17 interviews are summarized in this section. The findings are divided by the version of the categories that had the issue.

1. Current Version Issues

There were problems with the current questionnaire, but less than the revised questionnaire.

Respondents had trouble finding a category that fit the workers they hired. These include:

• Miscellaneous workers, human resources, and retail workers (one respondent thinks of his workers more along the lines of ‘full time’ and ‘part time’ instead of the work they were hired to do),

• Ag mechanics (they spend a lot of their time just working on maintenance of equipment and not necessarily driving the equipment. He ultimately put them under worker code 4 “Other Workers”),

• ‘Veggie preppers’ (he said that he would put them with field workers on the current form, but they don’t really fit that description very well),

• Pesticide applications (two respondents had trouble with this because this job takes too few hours to account for one “man-week” or “40 hours worth” since it is not a full time job. One respondent thinks that you are more likely to get it on the revised version since in the current version, it is lumped into category 4).

Some respondents had issues reading or understanding the categories, such as:

• One respondent classified all of his workers on the current version as livestock workers and combined them into code 2 (they were separate on the revised version as egg gatherers and inseminators). The interviewer thought this was incorrect and should have been 2 under livestock and 6 under “other”.

• As one respondent read the categories for the current questionnaire, he thought that the “jack of all trades and machinery operators seem different from pickers because one is skilled and one is unskilled” and did not see why they were together.

• Even though there are just four categories, at least one respondent did not initially see the category for supervisors/managers and had to go back to make changes when the interviewer was going through the debriefing questions.

2. Revised Version Issues

There were many problems that were specific to the revised version of the questionnaire. First, the table was initially intimidating to at least three respondents. One respondent found the length and detail daunting and said, “the more complicated, the more guessing.” That respondent also said that if he received the revised form of the questionnaire in the mail he probably would not answer it because of its length and the complexity. There were only two respondents who could fairly easily use the revised form with no apparent issues.

Next, navigation was difficult when the table was shown before the questions. At least five respondents who used the revised version first thought they needed to immediately do something with the table of categories and were confused where to go next – they started reading the categories, but didn’t know what to do with them. One respondent asked “how do you want me to answer?” when he looked at the category page.

Respondents had several problems reading the table of categories on the revised form. On his first read-through, one respondent saw all the field worker categories as ‘nursery and greenhouse’ and didn’t notice that they included ‘crops’. He figured this out, but not until he found nowhere to put his crop workers. Another respondent’s immediate response was “none of these apply to dairy – these are graders and sorters. We have people who milk cows and feed calves….where are those? Dairy isn’t included.” Then, as she read the categories, she wanted to put them in the “Animal Breeders” category because one of the examples for that category is “Dairy Husbandry Worker.” After reading ALL the categories (this took a lot of time), she decided on “All Other.” She said that they do not think of dairy cows as ‘livestock’ and we should have dairy separately. Interestingly, when she looked at the current categories, she immediately said “oh, there they are, livestock workers!”

At least half of the respondents who did the revised version first did not notice the broad categories on the left margin. Some respondents never saw the broad categories, while some noticed them once they couldn’t figure out where to categorize their workers, or when they moved to the current version. At several points in the interview, one respondent used the livestock categories to categorize his field workers (sorters and graders) because he didn’t notice that there were separate categories for each. He did not notice this until the very end of the interview. Another wanted to put her produce graders in category 22 with the livestock workers because she didn’t see the broad categories. Her business partner noticed this and moved them up to 13 under field workers. When asked how he might categorize his sons who do lots of different types of work, one respondent said either 21 or 12, but he wasn’t sure which one. When asked what he thought the difference was between those categories, he said that the 21/22/23/24 ones were “more specialized” than 11/12/13/14. He did NOT notice that one group was livestock workers and one group was field workers. One respondent considered using category 22 for one worker because he does a lot of grading and sorting, but then said “oh that looks like meat.” He did NOT see category 13 for crop graders and sorters at this point.

On the revised form, several respondents had difficulty finding a category for a specific type of work/worker. Examples include:

• truck driver,

• maintenance man,

• mechanic (ultimately put him in category 11 because he “worked on equipment”),

• someone who fixes equipment (would put in 11 or leave off),

• dairy milkers (maybe 42 or 21, but never confident),

• manure worker,

• hay balers (eventually used category 24 because “it says agricultural workers” - he did NOT see that it was livestock workers, or that it said “all other”),

• combine operator (eventually he put him in 11 because that category said ‘machine operator’),

• veggie preppers (never found a place on the current version, but in the revised version, he put them in category 23 for graders and sorters for livestock, because he didn’t realize that was for livestock),

• ag mechanic (put him under “other workers”)

• producing milk (never found a category),

• food safety (thought maybe 11),

• produce wrapper (he looked at the example “Carton Wrapper” for 14 hand packers and sorters and said ‘that makes no sense at all. Do you mean an egg carton wrapper? I wouldn’t include my sorters there because they are not egg wrappers,”

• workers who use equipment to apply pesticides (initially reported as “equipment operators” but then much later, the respondent saw “pesticide handlers and sprayers”. He said, “now look at that, pesticide handlers…but they are equipment operators.” He ultimately considered them equipment operators in category 11, but said that was an arbitrary decision),

• brother who runs the orchard and does all pesticide applications (when asked where he would categorize him, he said “out of all these? He does some of all of this.” He decided to put him under “11 because he does all of this (ag equipment operator)”.

• growing feed (never found a category), and

• family workers.

In addition to the examples given in III.A.3 (below) for both forms, respondents had more difficulty categorizing workers who did more than one type of work on the revised version because of the additional specific categories. These examples included greenhouse/field work, turkey inseminator/egg gatherer, livestock care/mowing pastures/driving tractors, field work/livestock work/animal breeding, netting/seeding, sod worker/landscape work, and sod worker/delivery truck driver. All of these categories would require just one code on the current version, but at least two on the revised version.

Part of the problem that respondents had was that they did not see the “other” categories. At least four respondents read through the entire list and did not seem to find the “All Other Workers” category.

In addition, at least three respondents had trouble distinguishing the categories from each other. One respondent asked “What’s the difference between farmworkers and agricultural workers?” (he didn’t see the words ‘all other’ at the end). Another said he thought some of the categories were overlapping and that was confusing. One respondent noticed that irrigation workers were listed in both code 12 and code 15.

One respondent had a very hard time with the new SOC codes. When he filled out the current form of the questionnaire, he said that most of his workers were fieldworkers, but when he went through the revised form, he did not place any workers under fieldworkers; he said he did not find any job descriptions in that general category that applied to his workers stating “so far I haven’t found anything I would put my workers in” after reading through all of the Field worker codes. He seemed a little perturbed that the breakdown of field workers (e.g., equipment operators) was not how they referred to the workers on his nursery operation. For example, he did not put workers in Code 11 because no one is hired specifically to work machinery – everyone uses the tractors and wagons. He did not put any workers in Code 12 because his workers do not manually plant. He seemed likely to put his workers under Code 15 until he read the specific examples under that code. Because the two examples did not apply to his workers, he rejected Code 15 also. He said he wouldn’t have even gotten to the Supervisors category before giving up, even though he reported workers under that category in the current form. This respondent’s problems are representative of the types of things interviewers heard for the revised version.

Respondents had mixed feelings and opinions about the descriptions and examples given for each category. Many liked the examples and specific descriptions, but interviewers noticed that respondents used the specific information to incorrectly categorize their workers. For example, one respondent said the examples helped her fill out the questionnaire, but she put her dairy workers under ‘animal breeders” because “dairy husbandry worker” was an example given there and none of the livestock worker categories had examples that were dairy related. Another respondent said that some descriptions and examples were helpful, but he mistakenly did not include his produce packers in the right category because the example of “carton wrapper” made him think it included only egg carton wrappers. Many respondents used the examples a bit too much which narrowed the category for them. If the examples did list a specific category that the respondent had on their operation, they were helpful. For example, egg gatherer and inseminator were examples listed for one respondent and those were two types of workers on his operation. In summary, the listed examples seem to be helpful when they contained examples directly relevant to the operation, but if the examples listed did not pertain to their operation, this was distracting and misleading to the respondent and resulted in incorrect data.

3. Issues with Categorizing Workers, Regardless of Version

Almost every respondent found it hard to categorize at least one worker on one or the other or both questionnaires. This problem had two components; difficulties because workers did more than one job and difficulties because respondents couldn’t find the proper category. Both difficulties were more problematic on the revised form.

Almost all respondents had workers that did more than one type of work. Examples included somewhat simple cases like “each worker has their primary responsibilities, but they all help each other out” to more complicated ones such as a team picking up irrigation pipes where one worker at a time drove equipment while the other picked up pipes. Other examples included overlap with dairy/crops/mechanical work/financial work/ bookkeeping, field work/milking cows, field work/pressure washing hog facility, and field work/supervising. In order to provide an accurate response, respondents would have to select more than one response for their workers on either version.

Respondents handled this problem in different ways. Some put workers in one category based on what their major work responsibility is, some put two codes in the answer box, one used the concept of man-hours or man-weeks, some used percentages, some filled out the table by person, some put the workers on the table twice (for example, one respondent listed 8 turkey egg gatherers and 6 turkey inseminators and 8 total workers) and some split their workers up across the categories to account for work that did not take up the majority of their time, but accounted for some non-trivial amount. This issue was more problematic on the revised version because there were more potential categories that a worker could be in.

The second issue was that respondents could not find an appropriate category for the type of work the worker did or was hired to do. For example, respondents could not find a place for workers who performed detasseling (they use no special equipment and are unskilled) and weeding, or who respondents described as researchers, electricians, maintenance workers, feed manufacturers, cleaning staff, harvest workers, mowers, and sprayers. This issue was complicated by the examples given on the revised form. Many respondents were looking for the very specific worker to be listed in the examples, but didn’t find it. Respondents handled this in different ways, including lumping those workers with the other workers they hire, excluding them from the form, writing a description in the table but not including a code, and including them in one of the ‘other’ categories. Interestingly, very few respondents chose to use those “other” categories.

4. Current vs. Revised Categories

Respondents were asked directly to give feedback on which form they preferred. That feedback is shown below.

1. Respondent preference for the current form

Four respondents preferred the current form, saying the revised version “takes more time,” “I had to read too much to find out where to put my guys,” and “who has time to read all this?” One thought that having fewer categories made you notice the catch-all category better.

2. Respondent preference for the revised form

Three respondents like the revised version better:

• One interviewer said one respondent “was a big fan of the more detailed descriptions. He felt that they were useful in helping him place his workers.”

• One respondent liked the revised version better saying that it is more explanatory.

• One respondent lumped together all his workers on the current form even though there are two distinct sets of workers with different wage rates (and also potentially working different numbers of hours) – one higher paid and with more skills, and one lower paid. In the new form, they fall in different categories so he reported them on separate lines.

Although some respondents said they liked the revised form’s descriptions and examples, they caused the respondent to inaccurately report their workers.

3. Mixed preference

Nine respondents had mixed reactions to the two versions:

• Three respondents had no preference for either version. The revised version took all three longer, but they could easily fit his workers into the categories. However, one did say “You used to have 4 and now you have 12 or 14? Wow, OK….” when he started the revised version (he did the current version 1st). Two of the three thought the current version is easier, but the revised version is more appropriate and more accurate.

• One respondent thinks the revised version would take longer but is easier because it has ‘more examples’.

• Four others thought the current form is easier because there are not so many choices, has less information and not so many ‘such as’ examples. But, they also thought the revised version is probably “more accurate” and “more comprehensive.”

• One respondent preferred the shorter categories because it was less reading but he thought that he was better able to place his workers in the categories due to the examples given.

5. Other Issues with the Ag Labor Questions (Not the Categories)

Several issues came up during the interviews that were not related to the categories, but other parts of the questionnaire.

Respondents did not always start at the right place in the questionnaire. When given the revised version first, at least 3 respondents never read the question about whether they had hired labor. One respondent went straight to the answer cell table (the interviewer assumed she went to the table because of the white space), one jumped to the include statement instead on question #2 and thought that was the question.

Five respondents had trouble determining the hours and wages paid. All five pay either twice a month or biweekly, making it difficult to determine pay for one week. One said that she would put 0 for wages, while four said they would just take ½ of a ‘normal’ paycheck that is given for a two week’s salary for each employee and ½ of the ‘normal’ hours.

At least 3 respondents were confused by the item codes. One saw the item codes and thought they were workman’s comp codes and that he had to report by those codes. The item codes confused two other respondents with one saying “I don’t know what a code of 411 means.”

Some respondents do not like the words “Supervisor” or “Manager”.

• Two did not like the wording of “manager” because it sounded like someone who was “always in charge.” Their workers are more like “working managers” because they all do a lot of work in addition to supervising. They wouldn’t be comfortable putting any of their workers in that category even if they do supervise or manage other workers.

• Another respondent is not comfortable with the term ‘supervisor/manager’ because he feels that the terms supervisor and manager imply that “they get paid more”. He also said that he and the other family members do a lot of other work such as running the orchard, working on machinery, running the wholesale business, giving shots to livestock, taking care of livestock, etc. He would not consider this supervising.

• Based on the interviewer’s personal knowledge of the operation, he thought one respondent placed the supervisors in categories 31 and 32 incorrectly. Out of the three total supervisors, the respondent placed 2 of them in 31 and 1 in 32. Since they are all making day-to-day decisions on the operation, the interviewer believed that all 3 should have been placed in category 31. There are also a couple of more senior workers who the interviewer believed should have been placed in category 32, but the respondent said that he would not classify them as “supervisors” because they do not have hiring and firing abilities.

One interviewer suggested the possibility of adding an example row to the table (shaded like we do in our other questionnaires). After the interviewer suggested this possibility, one respondent agreed that it would be helpful for reference purposes since he had some difficulty understanding that the codes from the box above needed to be inserted into the response cells below. At least two other respondents had trouble understanding that they were supposed to put the code from the job description box into the Worker Code column. An example row similar to those we use in other tabular question layouts may help to alleviate this issue.

Some respondents had trouble with the words “hired to do.” At least two respondents did not seem to understand this concept because they and their workers or family members do many types of work and are not “hired to do” any one thing. After discussion of the words “hired to do,” at least one respondent changed where he categorized his workers.

One respondent had larger concerns about the survey. Within a couple minutes of starting the interview, one respondent stated that this “was not going to be accurate”, because no one is spending the amount of time on these surveys to have accurate data. He then immediately brought up the Census of Agriculture and said that “it would take 3 days to fill it out correctly”.

There were lots of questions on what to include as hired workers. At least six respondents had at least one person that they weren’t sure about (children, grandchildren, parents, other relatives). Family workers, whether paid or unpaid, caused respondents problems. For example, one respondent would not include his mom who does the books because she is not paid.

Respondents had many questions and concerns about how to report their data.

• One respondent asked “am I reporting a weighted average? For example, I have 4 guys with 30 % of their time in irrigation. Should I report 3 guys as farm workers and 1 to irrigation? Or put all in 4 farm workers?” He ended up using a weighted average based on ‘man-weeks’.

• One respondent started to fill out the table, read the words outloud, including “work hired to do” and then wrote down one worker’s name on each line, When he got to the column for the “number of paid workers,” he was confused, and said “Oops, I was supposed to do this differently….ok.” He went back and wrote “tractor drivers” under code 11, and “watering” under code 12.

• At least two other respondents listed the individual people in the table, not the job.

• At least one respondent had trouble deciding which part of their operation we were asking about. It was a sod operation that had a production component and an installation component.

• One respondent was confused about whether we wanted to know about hours worked and wages by worker or by category of work. For example, should a worker be listed twice (in two rows) if s/he does two different types of work?

Many respondents had other concerns, including comments about other NASS surveys as well as agricultural issues. Those comments will be passed to Survey Administration Branch.

2. Second version summary

After the initial 17 interviews, we decided to make some changes to the category listing for the last few interviews to see if the changes made the revised version of the categories work better. The second version of the questionnaire is shown in Appendix C. The main changes that were made were a greater distinction between the categories, and the removal of the examples and detailed descriptions.

Seven additional interviews were done using the second version and those are summarized here. Two respondents wouldn’t technically do the cognitive interview with the interviewer, but one gave of the interviewers a breakdown of all his work categories (over 200) and the other had a general discussion with the interviewer.

There were no completely new problems in this round of testing, but similar problems to those we found in the first 17 interviews were identified. There were far fewer problems identified with the revised categories using the second version. In particular, respondents saw the broad category breakdowns. Additionally, because there were no examples for the categories, respondents could not get confused by them, nor could they use them to help categorize their workers. We did not see additional problems caused by the lack of examples. All respondents liked the current version better, but most could complete the second revised version.

1. Problems with the categories

• One respondent didn’t know where to start with the revised version since the category listing is on the left.

• At least three respondents had trouble categorizing workers because their workers do multiple jobs.

• One respondent asked what the difference is between first line supervisors and supervisors.

• There were also some difficulties categorizing particular workers:

  • Hay haulers (couldn’t decide between code 11 because it says ‘harvest crops’, code 12 because it says ‘loading harvested products’ and code 15).

  • Salesman

  • Supervisors - Under supervisors (code 31-Farmers, Ranchers and other Agricultural Managers), one respondent included 10 workers who ‘supervise visitors to the farm.” These are not coded the way we would want.

  • Office workers – the respondent forgot to include them on both forms

  • Pesticide workers – one respondent didn’t see this category on the revised version, but did on the current version. The interviewer assumes this is because there is so much less on the page in the current table.

• At least two respondents had a different accounting of their workers when using the two different forms, with both having more on the current form.

• One respondent needed 10 lines to record all her workers.

2. Problems with the questionnaire (not the categories)

• One respondent had trouble deciding what part of her operation we were asking about because she has many different pieces. She ended up answering for all pieces.

• One respondent answered for the whole year as she did not read the text.

• Work hired to do. One respondent suggested changing the name of the ‘work hired to do’ column to ‘type of work hired to do”.

4. RECOMMENDATIONS:

1. Recommendations related to the categories

On paper, move the category table AFTER the questions. Many respondents who completed the revised version first did not know what to do with the table when it came first. For CAPI and EDR, modifications need to be made so the person completing the form can see the categories and the questions at the same time.

Make the four original categories more obvious in the revised table (this change was implemented for the second version). Respondents did not see that many of the new categories in the first revised version were split up by livestock and field workers. The second revised version did not have this same problem. Ways to do this might include using a spanner across the top of each category, blank space between categories, larger font, color, upper case, taking out lines, grayscale and thicker lines. We suggest using the second revised version, which uses a spanner, blank space, and color (black and white). If the distinction between the four original categories cannot be implemented, then we recommend changing the names of the revised categories so that the field vs. livestock distinction is made first.

Change “Agricultural Workers, all others” to “All Other Agricultural Workers.” Many respondents couldn’t find a place for their workers because they did not see the “All Other” categories.

Remove examples or modify the way examples are given (this change was implemented in the second version). Respondents generally liked the additional information, but in many cases, the examples led the respondent to eliminate workers from the appropriate category. We cannot list ALL possible examples, so it may be better to eliminate the examples and provide a more general description. If we insist on examples, possible wording might be: “examples of workers in this category might be Combine Operator, Cotton Ginner, Hay Baler, etc.” At the very least, remove examples from categories that are self explanatory. For example, the “graders and sorters” category may not need examples.

Consider increasing the burden minutes for the survey. Although we did not have respondents complete the whole survey, several respondents said that the new version would take longer to complete.

It is possible that an example row in the answer table may help respondents understand how to fill out the table for their operation. NASS may want to review past research on such example rows in other surveys to determine how to format it and whether it will be effective.

We considered making the recommendation to move Supervisors to be the first category listed, as several respondents consider their supervisors as both farmworkers and supervisors. If that category was first, respondents may be more likely to use this category to account for them. However, while there is some evidence that this might help, there was no compelling evidence in our cognitive interviews that respondents had trouble ultimately categorizing their supervisors, and we may introduce more errors by moving it (respondents may be too likely to classify workers there who shouldn’t be because it’s listed first), so until further testing is done, we recommend leaving that category as is. We feel that making the distinctions between the broad categories more clear will help respondents answer correctly.

Virtually every respondent had at least one worker who did work in multiple categories of the revised questionnaire. The current version had some issues with this for workers who did both field and livestock work, but that is not as common as field workers who do multiple types or work within those subcategories or livestock workers who do multiple types of work within those subcategories. Most respondents could select one category based on a variety of different criteria (what the worker did the most that week, what they were hired to do the most that week, putting some of their workers in one category and some in a different category to reflect the percentage of work done in total, etc.). However, we anticipate that the new categories will cause more respondents to have trouble identifying ONE category for their workers. NASS recognizes that this issue may require more revisions and/or testing.

2. Recommendations unrelated to the categories.

Consider testing the understanding of the term ‘hired to do’. This term seemed to cause confusion for some, but not the majority of respondents, but since it is the basis of who to include in the questionnaire, it would be good to better understand how operators are interpreting it.

This survey instrument might be best administered with an interviewer on the phone or in person the first time an operation is sampled, particularly with the revised categories. Respondents had many problems figuring out how to answer the questions and where to categorize their workers.

We only performed cognitive interviews in person. We did not test the categories over the telephone. However, from the in-person cognitive interviews, I recommend that the CATI instrument first ask the original 4 four categories. After obtaining the numbers in each of those categories, then CATI interviewers should ask about the subcategories. I recommend this because of the length of the category listing as well as the difficulties respondents had noticing the separate groupings of field and livestock workers. Additionally, with the longer list on paper, many respondents never got to the bottom categories on the list, most importantly, the “all other workers” category. Also, consider how to put this survey on CAPI, since the new form requires the interviewer and respondent to see two pages at the same time. Both CAPI and EDR versions of the questionnaire should also be tested.

5. CONTACT PEOPLE

For questions or comments on the cognitive interviews, please contact Kathy Ott at 703-877-8000, x117 or [email protected] or Jaki McCarthy at 703-877-8000, x142 or [email protected].

For questions about the Agricultural Labor Survey, please contact Tony Dahlman at 202-720-7216 or [email protected] or Shiela Corley at 202-720-5921 or [email protected].

Summary of operations included in cognitive interviews

Table 2: Operations in Round 1 (used the first version of the questionnaire)

State	Type of Operation	Version used first	Number of hired workers	Other information
MD	Family run dairy	Revised	Up to 25 part time
MD	Family run dairy	Current	15 full time people all year
MD	Dairy	Revised	R said he has no hired labor (although he reported one hired worker on the April survey),	Asked about people who work for him that he does not consider ‘hired labor’ (his brother, two sons, grandson, mechanic).
MD	Orchard and vegetable grower	Current	Only recorded one, but has four families involved and hires many H2A workers that he did not count because they are hired through a contract
MD	Nursery	Current	Roughly 50 hired workers	Roughly 120 acres, 1.5 million in sales
MD	Livestock	Revised
MN	Seed corn and soybean	Current		Operates with two of his brothers
VA	Poultry	Revised	Approximately 8
VA	cattle/grain operation	Current	Father/son operation with no other paid employees	Primarily a cattle operation, but spend a significant amount of time on crops and hay.
VA	Apple grower	Revised	Hires 20+ workers
MO	Farrow to finish	Current	Approximately 22	Own 4000 sows
MO	Nursery	Current	Approximately 50	Includes a greenhouse and retail store.
MO	Research facility	Current	Approximately 97	Research facility, so workers hired for feed and care for livestock, as well as to conduct research
MO	Dairy	Current	Approximately 6	200 head dairy farm
MO	Hog, few beef cattle	Current	Approximately 4
WA	Sod	Current	Approximately 14	Part of a complicated establishment that also has a delivery company and a landscaping company.
WA	Orchard	Revised	Employs approximately 5 workers all year

Table 3: Operations in Round 2 (used the second version of the questionnaire)

State	Type of Operation	Version used first	Number of hired workers	Other information
GA	Grain, Wheat, Soybeans, Hay	Current	Varies, but 7 during the week we asked about.
GA	Nursery	Current	Approximately 31
GA	Nursery	Revised	3 workers at this slow time
GA	Fruit and Vegetable Processor	Current	Only 1 for Ag Labor survey, the rest are under “processing”	Wouldn’t fill out form, but engaged in conversation
GA	Nursery	Revised	Approximately 50-100	Wouldn’t really fill out form, but provided a listing of all the jobs done at his operation (200+)
GA	Blueberry Operation	Current	Approximately 5
VA	Produce	Revised	Approximately 31 part time	Operation has multiple pieces, including petting zoo, Christmas trees, pick your own produce, produce stand.

Cognitive Interview Protocol

In addition to the questions below, the full protocol used by cognitive interviewers contained background information, tips on recruiting participants, guidance on what to include in their summary, and an attachment that contained the protocol questions with enough space to take notes as the interviewer conducted the interview.

In the protocol, information for the interviewer is in regular text while things we expected the interviewer to say to the respondent are in italics. A version of the protocol to use during the interviews (with space for notes) was attached at the end of this document. The cognitive interview basically consisted of two think-aloud interviews – one using the current labor categories and one using the revised labor categories. Cognitive interviewers conducted about half of their interviews using the current version first and half with the revised version first.

Cognitive Interview protocol questions:

Introduction: Over the next several months, NASS will change the groupings used to categorize farm workers. This is the first time we are using these categories with farm operators, so we would appreciate your reaction to them. We are asking farm operators like yourself to review and test the questionnaire with us. There are no right or wrong answers for this process, and we welcome all of your feedback.

Part 1: First, please fill out this (current/revised) version of the form. Please fill it out as if you received it in the mail for your operation for July. As you answer the questions, please read and think aloud, describing the thought process you use to answer the questions. Throughout the interview, I may interrupt you to ask about particular issues or problems that you have. Again, there are no right or wrong answers for this process, and we appreciate your reaction to the categories.

Think aloud interviews are very hard for respondents to do. Throughout the interview, use these probes whenever the respondent does not think or read aloud and/or wherever appropriate:

• If the respondent asks for help or an explanation, say “how would you answer if I weren’t here?”

• Can you tell me in your own words what you think this question is asking? How did you arrive at that answer?

• If the respondent appears to be confused, ask: What made that question/category confusing for you?

• How could that question/category be improved? Are there other terms you would use to describe this type of worker?

Part 2: Now, please fill out the other version of the form. Again, please fill out this version of the questionnaire as if you received it in the mail for your operation for July. Please continue to read and think aloud, describing the thought process you use to answer the questions. Again, there are no right or wrong answers for this process, and we appreciate your reaction to the categories.

Use the same probes as above.

Part 3: After the respondent fills out both questionnaires, ask the following:

• In general, what types of work did you hire workers to do on your operation in July?

• (current version only) What types of work did you include in the:

  • field workers category?

  • livestock workers category?

  • supervisors/managers category?

  • other workers category?

• Are there any workers that you could not assign to a category? If so, what did you do on the form? What type of work do those workers do?

• Did you have any other issues fitting your farm workers into the categories on this form?

• Do any of your workers do work that they were not hired to do? For example, a worker was hired to pick vegetables, but ended up working in the greenhouse?

• Do any of the paid workers on your operation do more than one type of work? If so, how did you record these workers on the questionnaire?

• (revised version only) Please look at the descriptions and examples given for each category. Did they help you decide which workers fit into that category? Should the descriptions be shorter or longer? Are there other examples that you would include for any of the categories?

• Do you have any comments on the questions or categories that I didn’t cover?

Part 4: To wrap up:

• Did you find one form or the other easier to fill out? Which one? Why?

• Do you have any comments on anything that I didn’t cover yet?

14