Supporting Statement A (1220-0141)

BUREAU OF LABOR STATISTICS

OMB CLEARANCE PACKAGE

for

CLEARANCE TO CONDUCT COGNITIVE AND PSYCHOLOGICAL RESEARCH

IN FY2006 THROUGH FY2008

Prepared by

BEHAVIORAL SCIENCE RESEARCH LABORATORY

OFFICE OF SURVEY METHODS RESEARCH

BUREAU OF LABOR STATISTICS

November 17, 2005

Request for Revision to Clearance to Conduct

Cognitive and Psychological Research

Abstract

This is a request for clearance by the Bureau of Labor Statistics' (BLS) Behavioral Science Research Laboratory (BSRL) to conduct research to improve the quality of data collection by examining the psychological and cognitive aspects of methods and procedures. BLS staff employing state-of-the-art cognitive psychological testing methods will conduct these research and development activities. The feasibility and value of this approach to questionnaire construction, survey technology and interview processes has been demonstrated over the past 15 years. The use of this technique to improve the quality of data collection has been advocated by the Cognitive Aspects of Survey Methodology seminar sponsored by the National Academy of Science and participants in a questionnaire design advisory conference. The planned research and development activities will be conducted during FY2006 through FY2008 with the goal of improving overall data quality through improved procedures.

Supporting Statement

A. Justification

1. Collection of Information

The Bureau of Labor Statistics' Behavioral Science Research Laboratory conducts psychological research focusing on the design and execution of the data collection process in order to improve the quality of data collected by the Bureau. BSRL conducts research aimed at improving data collection quality by assessing questionnaire/form management and administration, as well as issues which relate to interviewer training and interaction with respondents in the interview process. BSRL staff work closely with economists and/or program specialists responsible for defining the concepts to be measured by the Bureau of Labor Statistics' collection programs.

Both questionnaires and forms are used in the Bureau's surveys. Questionnaires specify the preferred wording of the questions to be asked, whereas forms specify the data items to be collected. Each possesses distinctive problems which in many cases can be related to respondent characteristics, survey content, or format of administration. Such problems impede the effectiveness of particular surveys and the mission of the Bureau in general.

The purpose of this request for clearance for cognitive psychological research and development activities by the BSRL is to enhance the quality of the Bureau's data collection procedures and overall data management. The basic goal of the BSRL is to improve through interdisciplinary research the quality of the data collected and published by the BLS. BLS is committed to producing the most accurate and complete data within the highest quality assurance guidelines. It is with this mission in mind, then, that BSRL was created to aid in the effort of not only maintaining, but also improving the quality of the data collection process.

This laboratory was established in 1988, by Commissioner Janet Norwood, to employ behavioral science to investigate all forms of oral and written communication used in the collection and processing of survey data. This exploration also includes all aspects of data collection such as mode, manuals, and interviewer training. BSRL performs a state-of-the-art service to numerous programs within BLS, the DOL, and other agencies as requested, providing questionnaire redesign efforts, survey updates, and improvement in the overall quality of the data collection management. These efforts, in turn, increase data quality and reduce respondent burden. The techniques proposed here have been applied successfully to a number of on-going BLS surveys; e.g., Current Population Survey (CPS), Current Employment Statistics (CES), Consumer Expenditure Survey (CE), and the American Time Use Survey (ATUS).

The research techniques and methods to be used in these studies will include both analyses of questionnaire construction and interview process, as well as survey technology. Within the structure of the questionnaire, analyses will be conducted in the following domains:

1. Question Analysis--Evaluation of individual questionnaires appraising question intent, assessment of semantic clarity, and an examination of relationships between questions.

2. Term Analysis--Evaluation of specific wording and phrases in terms of their psycholinguistic properties and an assessment of respondent interpretation of the meaning of these terms, at both the conscious and unconscious levels.

3. Instruction Analysis--Inspection of instructions for their semantic clarity, the degree to which they reflect the stated intention of investigators, ease of interpretation, and other considerations which may elicit unambiguous and appropriate answers or behaviors from respondents or interviewers.

4. Format Analysis--Review of questionnaires or subsets of questions for perceptual characteristics in order to facilitate better respondent comprehension and promote more active attention to the focus of the questionnaire or form.

Within the interview process, several analyses are conducted to assess nonverbal communication, interpersonal dynamics and symbolic interaction--the use of cultural symbols to make social statements. Staff conducts research to evaluate the overall effectiveness of data collection procedural characteristics, including:

1. Interviewer Characteristics and Behavior analysis - Study of the presentation of e.g., appearance, manner, relation to subject population, etc., in order to enhance interpersonal skills of interviewers in general and develop and improve procedures for the training of interviewers.

2. Respondent Characteristics and Behavior analysis - Assessment of the social, cultural and ethnic characteristics of the respondent and how that may bear upon interactions with the interviewer. Staff members also observe the behaviors of respondents for cues concerning their reactions to the interview process. Because BLS constantly collects data from different populations that change over time, the analysis of respondent characteristics needs frequent updating.

3. Mode Characteristics - examination of the unique properties of interviewer and/or respondent behavior as a function of the media used to collect data; for example, self-administered interviews, personal interviews, telephone interviews and interviews utilizing assistive technologies (e.g., CAPI, CASI and CATI).

2. The Purpose of Data Collection

The purpose of BSRL's data collection is to improve Federal data collection processes through scientific research. Theories and methods of cognitive science provide essential tools for the development of effective questionnaires. For example, they can provide an understanding of how respondents comprehend survey questions, recall relevant facts from memory, make judgments, and respond. On the basis of such knowledge, questions can be tailored to increase the accuracy of the collected information and to reduce the respondent burden. Similar improvements can be made with respect to other aspects of the data collection process.

BSRL’s research contributes to BLS and to the entire survey research field. Research results are shared with the Bureau through seminars, training sessions, reports, publications, and presentations at professional conferences. The BSRL Staff has instituted a method of peer review to encourage high standards of social science research practice. A current bibliography of BSRL staff publications can be found in Attachment I.

The BSRL’s research is expected to 1) improve the data collection instruments employed by the Bureau, 2) increase the accuracy of the economic data produced by BLS and on which economic policy decisions are based, 3) increase the ease of administering survey instruments for both respondents and interviewers, 4) increase response rates in panel surveys as a result of reduced respondent burden, and 5) enhance BLS’s reputation resulting in greater confidence and respect in survey instruments used by BLS.

The application of cognitive and psychological theories and methods to survey data collection is widespread and well established. The consequences of failing to scientifically investigate the data collection process is to lag in the use of accepted practices, to apply archaic survey development techniques based on intuition and trial and error, and ultimately to incur a severe cost in data quality and in burden to respondents, interviewers and data users alike. For example, without knowledge of what respondents can be expected to remember about the past and how to ask questions that effectively aid in the retrieval of the appropriate information, survey researchers cannot ensure that respondents will not take shortcuts to avoid careful thought in answering the questions, or to be subject to undue burden. Likewise, without investigation of the interviewers’ roles and abilities in the data collection process, survey researchers cannot ensure that interviewers will read their questions correctly with ease and fluency, or record the respondent’s answers correctly.

3. Use of Improved Technology

Staff members will design, conduct and interpret field and laboratory research that contributes new knowledge of the cognitive aspects of human behavior in relationship to questionnaire design and survey methodology. Cognitive psychological research methods in use include such techniques as: probing questioning, memory cueing, group discussion, and intensive interviewing. Depending on research goals, these methods may be used separately or in combination with one another.

The use of the laboratory approach has a number of advantages associated with it. These advantages include: rapid and in-depth testing of questionnaire items, a more detailed understanding of the respondents’ comprehension of concepts, and access to special populations who can be quickly recruited and tested. Different laboratory methods will be used in different studies depending on the aspects of the data collection process being studied. Computer technology will be used when appropriate to aid the respondents and interviewers and minimize burden.

Respondent burden in this collection will be held to a minimum. The proposed approach to research of data collection methods is designed to obtain the maximum amount of information for the minimum respondent burden. The research includes such methods as:

1. interview pacing and latency classification,

2. degree of structure within the interview format, group dynamics observation and recording of decision behaviors and/or the negotiation processes,

3. structured tasks: card sorts and vignettes,

4. expert analyses, and

5. experiments involving the administration of forms to study respondents.

6. usability tests of existing or proposed data collection and data dissemination systems

4. Efforts to Avoid Duplication

This research does not duplicate any other research effort being done within the BLS. This research will provide critical, groundbreaking, and important supplemental information beyond that currently available in the field of survey methodology as it applies to BLS surveys.

This research also does not duplicate any outside-of-government research effort, as its purpose is not to replicate survey research studies. The staff of BSRL is cognizant of current research being done in the field of cognitive psychology through attendance at conferences, research reported in professional journals, and through in-house staff meetings and peer review processes. There is no current similar existing data that can be used or modified for the purposes of improving the overall data collection process.

5. Collection of Information Involving Small Establishments
   
   

BSRL data collection efforts focus primarily on information gained through laboratory interviews, telephone interviews, and self-administered questionnaires with individuals recruited from the general public. However, in some instances organizational goals necessitate the involvement of businesses, state agencies, or other entities. To the extent these establishments are included they normally are surveyed only once. Typically, this involves fewer than ten establishments per study, but when resources allow, additional establishments may be included to help generalize findings to the field.

6. The Consequences of Less Frequent Data Collection

The planned collection of data will allow BSRL to suggest modifications and alterations to survey research in an on-going manner. Because this collection is expected to be an on-going effort, it has the potential to have immediate impact on all survey collection methods within the Bureau's jurisdiction. Its delay would sacrifice potential gain in survey modification within the Bureau as a whole.

7. Special Circumstances

There are no special circumstances.

8. Federal Register and Consultation Outside BLS

Federal Register: No comments were received as a result of the Federal Register notice published in Volume 70, No. 172 on September 7, 2005

Outside Consultation: Consultation with individuals outside BLS to obtain views on the availability of data, frequency of collection, suitability of particular laboratory methods, clarity of instruction and record keeping, disclosure, or reporting format, and on the data elements to be recorded and/or disclosed, are frequent and on-going with the National Center for Health Statistics, the Bureau of the Census, the University of Maryland, the University of Michigan, and other federal agencies and institutions of higher learning. Consultants come from a wide range of subject areas and expertise. A list of individuals consulted in the past is attached to this document. (Attachment II).

The individual responsible for the BSRL research efforts is:

Dr. William P. Mockovak

Director of Behavioral Science Research

Office of Survey Methods Research

Bureau of Labor Statistics

PSB Room 1950

2 Massachusetts Ave., NE

Washington, DC 20212
202--691-7414

9. Payment to Respondents

For some research projects, lab staff will travel to, and test, in the vicinity of respondents' residences. In other cases, subjects will travel to the BSRL laboratory facilities. Because respondents are asked to leave their homes and travel to the laboratory field testing site, they will be reimbursed $35.00 for their time in the lab (approximately 1/2 hour to 2 hours) and $5.00 for transportation. With the exception of remuneration, there are no circumstances that require data collection to be inconsistent with 5 CFR 1320.5.

10. Confidentiality and Privacy Concerns

The data collected from respondents will be tabulated and analyzed only for the purpose of evaluating the research in question. Laboratory respondents will be asked to read and sign a Consent form explaining the voluntary nature of the studies, the use of the information, that the interview may be taped or observed, and a Privacy Act statement. (Attachment III). Surveys with current OMB approval that are involved in BSRL studies and are collected outside the laboratory such as mail or CATI surveys use the pledge of the existing approved collection or the Privacy Act statement.

The Commissioner's Order, "Confidential Nature of Bureau Records," explains the BLS policy on confidentiality: "In conformance with existing law and Departmental regulations, it is the policy of the Bureau of Labor Statistics that data collected or maintained by, or under the auspices of, the Bureau under a pledge of confidentiality shall be treated in a manner that will assure that individually identifiable data will be accessible only for statistical purposes or for other purposes made known in advance to the respondent."

11. Sensitive Questions

There are no questions of a sensitive nature.

12. Estimated Respondent Burden

The current OMB inventory for FY2005 is 3,000 hours. The FY2006, FY2007, and FY2008 estimated respondent burdens as follows:

Response Burden

(Hours)

FY2006 1,200

FY2007 1,200

FY2008 1,200

Total FY06-08 3,600

The burden hours are estimated based on the anticipation that the research will require approximately one hour per respondent. Each study will differ substantively from one another. The projects are expected to be complex, involving at times several cognitive testing methods to test the hypotheses of the given research question.

Coverage of the Estimates

The estimates cover the time that each respondent will spend answering questions, including the debriefing procedure concerning the cognitive testing procedure used. The time required to travel to the laboratory, if needed, is not covered, since distances and modes of transportation are unknown. No retrieval of information by respondents is anticipated, although it is possible that validation of data at some point may require respondents to keep and check records. In this case, experiments will be designed to include retrieval time.

Basis for the Estimate

These estimates are based on the BSRL’s previous experience in conducting such research under the existing OMB Clearance 1220-0141, and on expectations concerning the research projects to be conducted in the next 3 years. BSRL staff and its laboratory facilities (especially the usability lab) have been increasingly utilized by both program offices and outside agencies, and it is anticipated that this trend will continue. The estimates are also based on the experience of other government agencies (such as the National Center for Health Statistics' study of the Cognitive Aspects of Survey Methods 1987) which have conducted cognitively oriented questionnaire design research.

Annualized Cost to Respondents

Using the $35 per hour figure in number 9, the annualized cost to the respondents for laboratory time will be $42,000 for FY2006 – FY 2008 (based on 1,200 burden hours).

13. Total Annual Cost Burden

1. There will be no total capital and start-up cost component for respondents or record keepers resulting from the collection of information.

b) The respondents and record keepers will have no expenses for operation and
maintenance or purchase of services resulting from the collection of information.

14. Cost to the Federal Government

The maximum cost to the Federal Government is $48,000 annually for FY2006, FY2007, and FY2008. Those costs are entirely comprised of the $35 reimbursements and $5 travel allowance provided to respondents, and newspaper advertisement costs. Other costs such as operational expenses (e.g., equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without the paperwork burden, are in place as a function of the laboratory proper and is not contingent or necessary to perform this research. The overall annualized dollar cost to the respondents is $42,161. This assumes that approximately four-fifths of the participants will be reimbursed at a rate of $40 per session. The remainder will be volunteers willing to participate without reimbursement. The estimate dollar cost for these individuals is based on average hourly earnings of employed wage and salaried workers, $15.67 per hour, taken from Current Employment Statistics Program data.

960 respondents X $40.00 = $38,400

240 respondents X $15.67 = $3,761

15. Changes in Burden

This is a request for extension to the existing OMB Clearance 1220-0141 in order to continue the research mission for another 3 years. The estimated reduction of 1,800 hours results from a reduction in the projected number of respondents. Previously, BLS estimated that 4,000 respondents annually would participate in studies approved under this clearance; however, experience over the past three years indicates that the annual number of respondents is approximately 1,200.

16. Tabulation, Analysis and Publication Plans, and Project Schedule

This clearance request is for survey methodological and questionnaire quality assurance, which include the exploratory activities leading to the evaluation of data quality. Both quantitative and qualitative analyses are planned for the evaluation of these activities depending on the circumstances.

The results of these investigations will be used primarily to improve the quality of data collection and assure total collection quality as it relates to data management. Because BLS is using the latest techniques and cognitive psychological testing methodology, papers may be written that include some tallies of response problems, recall strategies, or results from other testing procedures used, etc.

Project Schedule

This project schedule calls for laboratory interviews to commence once OMB approval is received.

A time schedule is proposed that is continuously on-going in nature, with research publication dates dependent on data collection limited to the researcher's proposal and subsequent results.

17. Expiration for OMB Approval

The BSRL is not seeking approval to avoid displaying the expiration date for OMB approval of the information collection.

18. Exception to Certification Statement

There are no exceptions to the certification statement identified in Item 19 “Certification for Paperwork Reduction Act Submissions,” of OMB Form 83-I.

10