NASS Review

6

Brian Richards

November 10, 2009

My Comments: OMB 0596-0010 (Survey Proposals – U.S. Forest Service)

I. Introduction: In early November 2009, I received a request to review some survey proposals associated with U.S. Forest Service (USFS) planned surveys. I was forwarded 10 documents relating to this overall proposal. I list the ten documents below, in the order that they appeared (as attachments) on the 11/5/09 email message that I received. It will be helpful to provide a short hand reference for each document, since I refer to specific documents in sections II. and III. below.

(1) Firm Questionnaire 1 (unspecified state) – in the sections below, I will refer to this as “Firm 1.” (2) CA Firm Questionnaire 1 – in the sections below, I will refer to this as “CA Firm 1” (3) CA Firm Questionnaire 2 – in the sections below, I will refer to this as “CA Firm 2” (4) CA Firm Questionnaire 3 – in the sections below, I will refer to this as “CA Firm 3” (5) WY Firm Questionnaire – in the sections below, I will refer to this as “WY Firm” (6) Fuel wood (Household) Questionnaire – in the sections below I refer to this as “Fuel wood”

(7) GA Firm Questionnaire – in the sections below, I will refer to this as “GA Firm” (8) Firm Questionnaire 2 (unspecified state) – in the sections below, I will refer to this as “Firm 2.” (9) Supporting Statement – in the sections below, I will refer to this as “SS” (10) Firm Questionnaire 3 (unspecified state) – in the sections below, I will refer to this as “Firm 3”

Here are a few general observations about these ten documents. Nine of the ten documents referenced above are questionnaires; with the other document being the supporting statement, which will likely also be closely reviewed by OMB. Of the nine questionnaires, eight of them relate to firms, and one of them relates to households (the Fuel wood questionnaire in this last case). As a general statement, some of the questionnaires appear to be very early drafts. In addition, there are questionnaires referenced in the SS that are apparently not included in the package that I received. <<<I have included a table of the forms that will be used in each region. These forms will be edited to have the correct State and Year for each survey.>>>

II. My Comments (Relating to the Questionnaires):

For the 9 questionnaires, only two of them have acceptable “burden” statements (they are GA Firm and Firm 2). The Fuel wood questionnaire has a burden statement, but it does not meet the “OMB standard”, as I understand it. The following six questionnaires do not contain any burden statement, in their present form: Firm 1, CA Firm 1, CA Firm 2, CA Firm 3, WY Firm, and Firm 3. <<> All forms now have acceptable the “burden” statements indicated .>>>

Eight of the nine questionnaires have no detailed instruction sheets. The only one with any type of detailed, specific instructions is Firm 1. The other eight questionnaires have either no instructions or only very general instructions. Even Firm 1, with its detailed instructions, may be inadequate in this regard. For example, the first page of the actual questionnaire associated with Firm 1 asks for “Number of Employees”. The detailed instruction sheet contains no clarification on how this number should be determined. (As an aside, the Firm 1 questionnaire does not specify a reference year.) Should this number be the number of employees as of December 31 of the reference year? Should it be the maximum number of employees at any time during the reference year? Should it be an average or typical number of employees during the reference year? I have a similar observation for “GA Firm” and “Firm 2”. That is, how should the question on “number of employees” be interpreted? <<<Only the forms that will be mailed had instruction sheets included. Others will be collected by trained State cooperators. These forms have been used, with little change, for decades. In many cases, the responding firms has a prior questionnaire that was filled out for a prior years canvass. Contact information is provided that allows the respondent to contact someone to get answers to any questions. All of the “Firm” questionnaires use “industry terms” that are known to the respondents.>>>

Here are some other examples of where additional instructions would be beneficial, and these relate to “CA Firm 1” In section II. of this questionnaire, the middle column of the associated table requests that the volume of raw material received be provided. The column simply states “specify units” in parentheses. Some additional instructions could provide some helpful clarification on what types of units would commonly be provided. As another example, question J. in section I. requests the number of “production workers” that worked at the facility in the reference year. Some additional instructions would provide some helpful clarification on what types of workers should be classified as production workers. For example, should this tally include only the actual line workers? Or should this tally also include shift supervisors that oversee the line workers? <<<Instructions and/or forms edited. >>>

Some questionnaires displayed a reference year, and some did not. But for those did provide a reference year, I have a few general concerns. For “GA firm”, the reference year is displayed as 2005. Will this be the actual reference year, … or did the USFS just choose not to update this for this early draft? If this is the actual reference year, I would have great concerns relating to the respondents ability to recall details from 2005. Similarly, for “Firm 2” the reference year is displayed as 2008. Again, if this is the actual reference year, I would have some concerns relating to the respondents ability to recall details from 2008 (given that we will soon be in 2010). <<<I have included a table of the forms that will be used in each region. These forms will be edited to have the correct State and Year for each survey. Data will be collected during the OMB approval period. The data information collected will be for the year prior to the year that data is collected.>>>

I have a general comment regarding these nine questionnaires. In my cursory review, I noticed many typos and grammar problems. I am assuming that the USFS will run these documents through a spell checker and grammar checker before sending their package to OMB. Some additional specific comments are as follows. In “Firm 1”, there is a section I. and II., as referenced in the instruction sheet. However, there are some questions on the page preceding section I. For formatting consistency, it might be helpful to display this early set of questions as section I., and then the existing section I. would become section II., and the existing section II. would become section III. <<<Corrected all typos and grammar found. The Sections were not changed. These forms have been the same format for decades.>>>

In the Fuel wood questionnaire, question 17 asks the respondent to sum different parts and subparts in different ways. I believe that the existing layout will cause confusion with the respondents. It was requested that 17A and 17B sum to 100%. In addition, it was requested that 17A1 through 17A3 sum to 100% also, as well as 17B1 through 17B4 sum to 100%. If, for example, 17A is displayed as 60%, then many respondents may have 17A1 through 17A3 summing to 60% also, not 100%. The USFS might consider an alternate way to ask for percentages in these specific questions. Regarding a somewhat similar issue, I would suggest that the USFS provide more details on how they would like respondents to display percentages in section V. of “GA Firm”. <<<Simplified section. Changed to only one set of numbers that needs to sum to 100%>>>

The “Firm 3” questionnaire totals two pages. There is much information / instructions / questions crammed into these two pages. Among other potential problems, I believe it could cause some problems when it comes time to enter responses into a data entry system. I might suggest to the USFS to reformat this, perhaps spreading over three pages or more, instead of the existing two pages. This would improve the “readability”, both for the respondents and those persons entering the responses into a data base. <<< The “Firm 3” form is an attempt to simplify the Standard TPO form “Form 1” and “GA Form”. It eliminates the need for the spreadsheet table found in Section I of “Form 1” and Section III of the “GA Form”. There are only 14 questions on the “Firm 3” questionnaire, including Address section, short Mill Info section, and a Notes section. This form has been reviewed by State cooperators and potential respondents>>>

III. My Comments (Relating to the Supporting Statement):

As I read through the 17 page SS, I was primarily interested in the relevant explanations associated with the statistical methodology. But I had a few other comments relating to more general survey and methodology issues. Section A.2.b. explains that “The Pulpwood Received Questionnaire is sent annually in a complete mail canvass … The form is completed by all mills … “. How “good” is our frame, relating to firms, which serves as a basis for our sampling universe? In a similar fashion, the response to this section also states that “ … Residential fuelwood and post information is collected from a random sample of all households in an individual state throughout the United States. … “ Again, I ask the rhetorical question, how “good” is our frame, relating to households, which serves as a basis for our sampling universe? <<< Pulpwood and other “Firms” questionnaire. Mills are canvassed based on an historical list maintained at each Region, Mill directories, Industry personnel, Industry publications, and State cooperators.

2) Residential fuelwood and post. Will change to: “…is collected from a random sample of all households in an individual state throughout the United States. The list of random households will be obtained from marketing/polling companies chosen by the State.”>>>

In a similar fashion, section A.2.d. explains that “new mills come on line and existing mills close temporarily or go out of business”. I assume that the USFS feels confident that, however they build and maintain their sampling frame, that the coverage of these mills is adequate. That is, I assume that USFS has resources or information that allows them to keep up with new mills coming on line. In the second paragraph of this subsection, it is stated that a particular questionnaire ” … is sent annually … to pulpwood processors in other countries that receive pulpwood from the United States.“ If we are dependent upon survey responses from firms from other countries, I am assuming that we obtain adequate response from these international firms? I am curious, how often does it occur that an international firm has difficulty reading and understanding a questionnaire that is in English? I am assuming that the USFS does not provide questionnaires in other languages, aside from English. <<<1) Mill closings and openings are tracked by USFS personnel, Mill directories, Industry personnel, Industry publications, and State cooperators. 2) The only international firms that are canvassed are Canadian Firms. English is understood >>>

Section A.3. addresses “ ... to what extent, the collection of information involves the use of … elec­tronic … or other techno­log­ical collection techniques … ”. In their response, the USFS stated that “ Due to … .. the access to e-mail addresses (or lack of access to?), and the lack of access some respondents may have to the internet, no improved information technology exists or is feasible to reduce burden … “ As a general comment, I would hope that the USFS is working towards implementing Electronic Data Reporting (EDR), even if just on a limited basis for now. As we approach 2010, I would think that many firms / households have access to the Internet. In addition, the USFS does not necessarily need the email addresses of individual firms / households to make it feasible for the firms / households to respond electronically. The questionnaire (and accompanying instructions) that is mailed to the sampled firms / households merely would need to contain instructions on how respondents access the appropriate Internet site, to submit their survey responses electronically. The technology would seemingly exist in this day and age. It would just seem a matter of USFS taking the steps to make the technology available to respondents. As is documented elsewhere, the Government Paperwork Elimination Act (which took effect in 2003) mandates federal agencies to provide electronic ways of doing business. <<<The Northern Region attempted to collect “Firm” information over the internet 3-4 years ago. A site was set up to collect information over the internet. What we learned was that the mills will not take the time to log into the internet, log into our site and fill out a survey online. The larger mills are more likely to use e-mail with a survey form attached. We canvass many small mills that are operated by only a few people that will not go on-line to respond to a voluntary survey, if they do have internet access at all. The Northern Region is currently considering trying to get a Mill Canvass site set up, but not till we can get a data entry site set up first. The data entry site will be used by USFS personnel and cooperators to enter in mill forms collected on paper. This is still in the planning stages and will be years down the road.>>>

Section A.12. addresses burden that will be incurred by the survey respondents. In their response, the USFS displayed partial information (Table 2), in that they showed the requested information for “Fuelwood”, “GA Firm”, and “Firm 2”. I am assuming that OMB would want to see this information for all the questionnaires, including the other six questionnaires that I referenced in section I. above. I also understand that the burden estimates should be given in terms of all the potential contacts that may need to be made (i.e., a firm / household may be a nonrespondent on the first and second attempt). The information provided in Table 3 is also displayed for only three of the questionnaires. Lastly, there seems to be a math error in arriving at the total dollar amount displayed in the bottom right cell of Table 3. <<<There are only 3 types of data that is being collected: Residential Fuelwood and Post, Pulpwood Received and Logs and Other Roundwood Received. The Northern, Southern, and Western regions use slightly different questionnaire to gather this information. I have included the questionnaires from all the regions as examples. Section A.2.a is updated to indicate which of the attached questionnaires is used for each of the canvases. I have included a table (new Table 1) that shows which State will be canvassed during which year and which questionnaire will be used.

Fixed error in Table 3 (now Table 4)>>>

Section B.1 addresses the need for the relevant agency to provide the number of units (generally speaking) in the universe, as well as the number of units they are planning to sample. The table that was provided in the response displays certain information for a household survey in MN (shown as Table 4). I am assuming that OMB would like to see this information for all surveys that are being planned – that is, for all the different surveys of households and firms that will be carried out in the various states. For Table 4 that is provided, the USFS displays a column for the population size (of households), as well as a column for the estimated number of respondents. I am assuming that in all such tables, that the USFS should also provide the actual planned sample sizes, in addition to possibly providing the estimated number of respondents. <<<Only the Residential Fuelwood and Post canvass is covered in Part B. The Pulpwood Received and Logs and Other Roundwood Received questionnaires are a canvass of all wood processing mills, and thus are not included in Part B.>>>

I reviewed section B.2. carefully, and had a number of comments and questions. The response from USFS mentioned that, for the household surveys, that households will be stratified into metro and non-metro strata. It was not clear, what variables would be used to stratify further – i.e., within the metro and non-metro components. In their response, there was no indication on how USFS plans to stratify for the surveys of mills / firms. I assume that this is information that OMB would like to have. In their response, USFS also stated that “ … about 1 out of 10 (firms) will be sampled in states with 500 or more firms”. This 1 out of 10 approach may not be the most efficient approach to use, if they were planning to use that across the board. If sampling 60 out of 600 firms would provide reasonable precision, then sampling 500 out of 5000 firms might be “overkill” (i.e., might provide more precision than would be necessary). In the existing response, a statement was made that was confusing to me; it was “For states that do not have a list of logging firms or firewood processors, all known logging firms or firewood processors will be canvassed.” If states did not have a list of firms, how did they know where to canvass to contact the firms? Or stating this differently: if states know where to go to canvass the relevant firms, then there must have been some type of pre-existing list (?). <<<changed section to answer questions>>>

One of the statements in the response to this section that was of particular concern to me was “Based on past experience, the response rate is expected to be around 30 percent.” As I understand it, OMB would like planned surveys to achieve a response rate of at least 80%. If it is projected that response rates will be significantly less than this, then the relevant agency should have a plan in place that would allow them to significantly increase response rates. From everything I read in the relevant responses, I did not see anything that would allow these surveys to come anywhere close to an 80% response rate. It was not clear if this projected response rate of 30% was for the household surveys or the firm-related surveys, or both. It also was not clear if this projected response rate was the overall response rate, after all follow-up was carried out ... or if this was the projected response rate after the initial mail-out? As a general comment, there is seemingly a lack of detail and clarity, relating to how nonresponse will be handled (operationally) for the various firm-related and household surveys. <<<Based on past experience, literature searches, and working with marketing/polling companies, all agree that 30% is the highest return you can expect for a voluntary mail survey. Even the U.S. Census, a “required by law” survey is only expecting a little over 60% response rate.>>>

The last pair of statements in the response to B.2. indicated a projection of 20% for the CV for two primary estimates – namely, “amount of fuelwood” and “posts cut”. From my review of the questionnaires, the amount of fuelwood would seem to be an important question on the household survey. However, for at least four of the firm-related questionnaires, “posts cut” does not seem to appear as a question asked. Questionnaires relating to Firm 1, Firm 2, Firm 3, and GA Firm relate to roundwood received, pulpwood received, etc. and apparently do not ask “posts cut”. It also was unclear what “estimation level” the target CVs related to. Since the surveys were state-related, I assume this CV either related to a state-level estimate, or some type of “regional” estimate (i.e., maybe a particular state had 4 to 8 regions defined?). If this CV target related to state-level estimates, then such estimates would seem relatively imprecise. I am assuming that OMB would have some concerns with such projected precision. Lastly, I am assuming that these projected CVs of 20% were realistic, in that they were consistent with CVs realized in past surveys? <<< The Pulpwood Received and Logs and Other Roundwood Received questionnaires are a canvass of all wood processing mills and are not included in Part B. The CV of 20% is based on the historical record of past Residential Fuelwood and Post Surveys and is realistic and achievable. . The CV of 20% is the target for the Forest Inventory Units in the State and not for each stratum within a Forest Inventory Unit>>>

The response to B.3. also was of concern to me. The very last statement of this one-paragraph response reads “In the event the total number of households canvassed is not reached, a random selection of non-respondents is mailed the questionnaire a second time.” From reading this, one can assume that, in general, there is no follow-up planned of nonrespondents? Is this for both the household and firm surveys? If the response rate will truly be as low as 30%, then it becomes very important what non-response adjustment (NRA) methodology will be carried out. With respect to statistical details, the USFS response did not provide any insight into how the NRA will be carried out. For the firm-related surveys, for example, perhaps the NRA will be carried out uniquely within a sampling stratum? If so, then it becomes very important how we ultimately choose to stratify (I emphasize again, that such details were not included in the SS). <<<Section explained better.>>>