SUPPORTING STATEMENT
U.S. Department of Commerce
U.S. Census Bureau
Generic Clearance for Questionnaire Pretesting Research
OMB Control No. 0607-0725
A. Justification
1. Necessity of Information Collection
In recent years, there has been an increased interest among federal agencies and others in the importance of testing questionnaires. This interest has been spurred by a recognition that the traditional methods of pretesting are weak tools for evaluating questionnaires and procedures. These methods consist of a small "hothouse" field test accompanied by interviewer debriefing, and the information collected through their use is quite limited in its ability to detect and diagnose problems with the instruments and the procedures being tested.
In response to this recognition, new methods have come into popular use, which are useful for identifying questionnaire and procedural problems, suggesting solutions, and measuring the relative effectiveness of alternative solutions. Through the use of these kinds of techniques, employed routinely in the testing phase of Census Bureau surveys, questionnaires can be simplified for respondents, respondent burden can be reduced, and the quality of the questionnaires used in continuing and one-time surveys can be improved. Thus an increase in the quality of the data collected through these surveys can be achieved as well.
In September 1991, the Census Bureau requested and received a generic clearance (Number 0607-0725) on an experimental basis, which relaxed some of the time constraints and enabled the Census Bureau to begin conducting extended cognitive and questionnaire design research as part of testing for its censuses and surveys. The clearance covered data collections in the demographic, economic, and decennial areas of the Bureau, and specifically applied to research that is focused on questionnaire design and procedures aimed at reducing measurement errors in surveys. Research on paying respondents was specifically excluded from the clearance. As part of the experimental clearance, the Census Bureau submitted to the Office of Management and Budget (OMB) a report that contained indicators of the work that was conducted under the clearance. At the end of the experimental period, the Census Bureau requested and received a three-year renewal of the clearance (through December 1995), covering the same kinds of research activities. As part of the clearance, the Census Bureau has submitted to OMB a report of pretesting activities at the end of each year of the clearance.
Subsequently, the Census Bureau has received five more three-year renewals of the generic clearance for pretesting (through August 2010). The current clearance contains approval for three additional types of activities: research about incentives, expanded field tests conducted to include split sample questionnaire experiments in multiple panels, and usability testing of electronic instruments.
At this time, the Census Bureau is seeking another three-year renewal of the generic clearance for pretesting, with the same conditions as the previous clearance. This will enable the Census Bureau to continue providing support for pretesting activities, which is important given the length of time required to plan the activities.
The specific methods proposed for coverage by this clearance are described below. Also outlined are the procedures currently in place for keeping OMB informed about the identity of the surveys and the nature of the research activities being conducted.
The methods proposed for use in questionnaire development are as follows:
Field test. For the purposes of this clearance, we are defining field tests as small data collection efforts of 500 cases or less, conducted among either purposive or statistically representative samples, for which evaluation of the questionnaire and/or procedures is the main objective and no plans to publish the data other than for purely methodological purposes are envisioned.
Field tests are an essential component of this clearance package because they serve as the vehicle for conducting standardized behavior coding of the interaction between the respondent and the interviewer. This methodology does not require any additional data collection above and beyond the field test–it involves applying a standardized coding scheme to the completion of a field interview, either by a coder using a tape-recording of the interview or by a "live" observer at the time of the interview. The coding scheme is designed to identify situations that occur during the interview that reflect problems with the questionnaire. For example, if respondents frequently interrupt the interviewer before the question is completed, the question may be too long. If respondents frequently give inadequate answers, this suggests there are some other problems with the question. Quantitative data derived from this type of standardized coding scheme can provide valuable information to identify problem areas in a questionnaire, and research ("New Techniques for Pretesting Survey Questions" by Cannell, Kalton, Oksenberg, Bischoping, and Fowler, 1989) has demonstrated that this is a more objective and reliable method of identifying problems than the traditional interviewer debriefing, which is typically the sole tool used to evaluate the results of a traditional field test.
Interviewer debriefing has advantages as well, since it utilizes the knowledge of the employees who have the closest contact with our respondents. In conjunction with other methods, we plan to use this method in our field tests to collect information about how interviewers react to the survey instruments.
Field tests conducted under this clearance will involve either purposive or statistically representative samples. Under this clearance a variety of surveys will be pretested, and the exact nature of the surveys and the samples is undetermined at present. However, due to the small nature of the tests, we expect that some will not involve representative samples. In these cases, samples will basically be convenience samples, which will be limited to specific geographic locations and may involve expired rotation groups of a current survey or census blocks that are known to have specific aggregate demographic characteristics. The needs of the particular sample will vary based on the content of the survey being tested, but the selection of sample cases will not be completely arbitrary in any instance.
Respondent debriefing questionnaire. In this method, standardized debriefing questionnaires are administered to respondents who have participated in a field test. The debriefing form is administered at the end of the questionnaire being tested, and contains questions that probe to determine how respondents interpret the questions and whether they have problems in completing the survey/questionnaire. This structured approach to debriefing enables quantitative analysis of data from a representative sample of respondents, to learn whether respondents can answer the questions, and whether they interpret them in the manner intended by the questionnaire designers.
Split sample experiments. This involves testing alternative versions of questionnaires, at least some of which have been designed to address problems identified in draft questionnaires or questionnaires from previous survey waves. The use of multiple questionnaires, randomly assigned to permit statistical comparisons, is the critical component here; data collection can include mail, telephone, or personal visit interviews or group sessions at which self-administered questionnaires are completed. Comparison of revised questionnaires against a control version, preferably, or against each other facilitates statistical evaluation of the performance of alternative versions of the questionnaire.
In any split sample experiments conducted under this clearance, alternative questionnaire versions will be tested. The number of versions tested and the number of cases per version will depend on the objectives of the test. We cannot specify with certainty a minimum panel size, although we would expect that no questionnaire versions would be administered to less than fifty persons in a split sample test.
Split sample tests that incorporate methodological questionnaire design experiments will have a larger maximum sample size (up to several hundred cases per panel) than field tests using other pretest methods. This will enable the detection of statistically significant differences, and facilitate methodological experiments that can extend questionnaire design knowledge more generally for use in a variety of Census Bureau data collection instruments. The Census Bureau will consult with OMB prior to submission regarding split sample tests with sample sizes over 1000.
Cognitive interviews. This method involves intensive, one-on-one interviews in which the respondent is typically asked to "think aloud" as he or she answers survey questions. A number of different techniques may be involved, including asking respondents to paraphrase questions, probing questions asked to determine how respondents came up with their answers, and so on. The objective is to identify problems of ambiguity or
misunderstanding, or other difficulties respondents may have answering questions. This is frequently the first stage of revising a questionnaire.
Usability Interviews. This method involves getting respondent input to aid in the development of automated questionnaires and websites and associated materials. A number of different techniques may be involved, such as one-on-one usability interviews with think aloud, probing, and paraphrasing tasks, card-sorting techniques, and disability accommodation testing. The objective is to identify problems that keep respondents from completing automated questionnaires accurately and efficiently, with minimal burden or that prevent respondents from successfully navigating websites and finding the information they seek.
Focus groups. This method involves group sessions guided by a moderator, who follows a topical outline containing questions or topics focused on a particular issue, rather than adhering to a standardized questionnaire. Focus groups are useful for surfacing and exploring issues (e.g., confidentiality concerns) which people may feel some hesitation about discussing.
Attachment A is a list of surveys for which work has been done under the generic clearance over the past three years, as well as the types of pretesting methods used.
Procedures for Clearance
This clearance will only cover pretests that involve more extensive testing than the traditional field test with interviewer debriefing as the only evaluative component. Since the types of surveys included under the umbrella of the clearance are so varied, it is impossible to specify at this point what kinds of activities would be involved in any particular test. But at a minimum, one of the types of testing described above or some other form of cognitive pretesting would be incorporated into the testing program for each survey.
We will provide OMB with a copy of questionnaires, protocols, and debriefing materials in advance of any testing activity. Depending on the stage of questionnaire development, this may be the printed questionnaire from the last round of a survey or a revised draft based on analysis of other evaluation data. When the time schedule for a single survey permits multiple rounds of testing, the questionnaire(s) for each round will be provided separately. When split sample experiments are conducted, either in small group sessions or as part of a field test, all the questionnaires to be used will be provided. For a test of alternative procedures, the description and rationale for the procedures would be submitted. A brief description of the planned field activity will also be provided. OMB will endeavor to provide comments on substantive issues within 10 working days of receipt.
Any large field tests or dress rehearsals that follow from the initial questionnaire development activity included here are not covered by this generic clearance. Separate submissions for any such data collection efforts will be made.
The Census Bureau will consult with OMB prior to submission on the appropriateness of submissions under this clearance that may raise policy or substantive issues.
The Census Bureau will send OMB an annual report at the end of each year summarizing the number of hours used, as well as the nature and results of the activities completed under this clearance.
Data collection for this project is authorized under the authorizing legislation for the questionnaire being tested. This may be Title 13, Sections 131, 141, 161, 181, 182, 193, and 301 for Census Bureau-sponsored surveys, and Title 13 and 15 for surveys sponsored by other Federal agencies. We do not now know what other titles will be referenced, since we do not know what survey questionnaires will be pretested during the course of the clearance.
2. Needs and Uses
The information collected in this program of developing and testing questionnaires will be used by staff from the Census Bureau and sponsoring agencies to evaluate and improve the quality of the data in the surveys and censuses that are ultimately conducted. None of the data collected under this clearance will be published for its own sake.
Because the questionnaires being tested under this clearance are still in the process of development, the data that result from these collections are not considered official statistics of the Census Bureau or other Federal agencies. Data will be included in research reports prepared for sponsors inside and outside of the Census Bureau. The results may also be prepared for presentations related to survey methodology at professional meetings or publications in professional journals.
Information quality is an integral part of the pre-dissemination review of the information disseminated by the Census Bureau (fully described in the Census Bureau's Information Quality Guidelines). Information quality is also integral to the information collections conducted by the Census Bureau and is incorporated into the clearance
process required by the Paperwork Reduction Act.
3. Use of Information Technology
When the census or survey being pretested employs automated methods for its data collection, the questionnaire research conducted under this submission will also utilize automated data collection techniques.
4. Efforts to Identify Duplication
This research does not duplicate any other questionnaire design work being done by the Census Bureau or other Federal agencies. The purpose of this clearance is to stimulate additional research, which would not be done under other circumstances due to time constraints. This research will involve collaboration with staff from other agencies who are sponsoring the surveys conducted by the Census Bureau. The research may also involve joint efforts with staff from other Federal laboratory facilities. All efforts would be collaborative in nature, and no duplication in this area is anticipated.
To the maximum extent possible, we will make use of previous information, reviewing results of previous evaluations of survey data before we attempt to revise questionnaires. However, this information is not sufficient to refine our census and survey questionnaires without conducting additional research.
5. Minimizing Burden
This research will be designed as relatively small-scale data collection efforts. This will minimize the amount of burden required to improve questionnaires and procedures, test new ideas, and refine or improve upon positive or unclear results from other tests.
6. Consequences of Less Frequent Collection
This clearance involves one-time questionnaire development activities for each survey that is connected with the clearance. If this project were not carried out, the quality of the data collected in the surveys would suffer.
7. Special Circumstances
All the guidelines listed in the OMB guidelines are met. There are no special circumstances.
8. Consultations Outside the Agency
Consultation with staff from other Federal agencies that sponsor surveys conducted by the Census Bureau will occur in conjunction with the testing program for the individual survey. Consultation with staff from other Federal laboratory facilities may also occur as part of joint research efforts. These consultations will include discussions concerning potential response problems, clarity of questions and instructions, and other aspects of respondent burden. Additional efforts to consult with potential respondents to obtain their views on the availability of data, clarity of instructions, etc., may be undertaken as part of the testing that is conducted under this clearance.
A notice was published in the Federal Register on April 2, 2010 (vol. 75 FR, pg. 16737), inviting public comment on our plans to submit this request. In response to this notice, we received one comment, which expressed strong support for pretest research for censuses and surveys to identify measurement problems with questionnaires and survey procedures
9. Paying Respondents
Respondents for activities conducted in the laboratory (that is, cognitive interviews and focus groups) under this clearance will receive a small stipend. This practice has proven necessary and effective in recruiting subjects to participate in this small-scale research, and is also employed by the other Federal cognitive laboratories. The incentive for participation in a cognitive interview is $40, and for participatiin a focus group it is $50 - $75 unless approval is granted by OMB on a case-by-case basis. Respondents for methods that are generally administered as part of field test activities (that is, split sample tests, behavior coding of interviewer/respondent interaction, and respondent debriefing) will not receive payment unless there are extenuating circumstances that warrant it.
10. Assurance of Confidentiality
All respondents who participate in research under this clearance will be informed that the information they provide is confidential and that their participation is voluntary. For personal visit and telephone interviews, this information will be conveyed verbally by the interviewer. For personal visit interviews, respondents will also be notified in writing so they will have something they can keep and read. For self-administered questionnaires, the information will be included in the mailing package, either on the questionnaire or the instructions. All participants in cognitive research will be required to sign written notification concerning the voluntary and confidential nature of their participation. We will also inform respondents in writing of the need to have an OMB number. Copies of the five generic clearance consent form templates are contained in Attachment B. They include a consent form for conducting demographic pretesting with adults, parental consent form for conducting demographic pretesting with minors, consent form for conducting usability interviews with adults, parental consent form for conducting usability interviews with minors, and a consent form for conducting economic pretesting.
The confidentiality of information is assured by Title 13, United States Code, or other applicable titles which authorize the collection of information.
11. Justification for Sensitive Questions
Most of the questions that are included on Census Bureau questionnaires are not of a sensitive nature and should not pose a problem to respondents. However, it is possible that some potentially sensitive questions may be included in questionnaires that are tested under this clearance. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and alleviate them insofar as possible before the actual survey is administered.
12. Estimate of Hour Burden
The total estimated respondent burden is 16,500 hours for the period from September 2010 through August 2013. These hours will be distributed as follows:
Respondent burden (hours)
September 2010 - August 2011 5500
September 2011 - August 2012 5500
September 2012 - August 2013 5500
Total 16500
This estimate is based on our experience with this clearance over the past 19 years, and our anticipated use of the clearance over the next three years, including burden hours expended for multi-panel questionnaire experiments.
A variety of forms will be used in conducting the research under this clearance, and the exact number of different forms, length of each form, and number of subjects/respondents per form are unknown at this time.
13. Estimate of Cost Burden
There is no cost to respondents for participating in the research being conducted under this clearance, except for their time to complete the questionnaire.
14. Cost to Federal Government
There is no way to anticipate the actual number of participants, length of interview, and/or mode of data collection for the surveys to be conducted under this clearance. Thus, it is impossible to estimate in advance the cost to the Federal Government. Costs will be covered by divisions conducting the research from their data collection budgets.
15. Reason for Change in Burden
There are no changes in burden. The current annual allotment is 5,500 hours.
16. Project Schedule
This research program is for questionnaire and procedure development purposes. Data tabulations will be used to evaluate the results of questionnaire testing. The information collected in this effort will not be the subject of any printed Census Bureau reports; however, it might be included as a methodological appendix or footnote in a report containing data from a larger data collection effort. The results of this research may, however, be prepared for presentation at professional meetings or publication in professional journals.
Due to the nature of this clearance, there is no definite or tentative time schedule at this point. We expect work to continue more or less continuously throughout the duration of the clearance.
17. Request to Not Display Expiration Date
No exemption is requested.
18. Exceptions to the Certification
There are no exceptions to the certification.
| File Type | application/octet-stream |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |