ACF Survey Pre-test SUPPORTING STATEMEN 10 08

SUPPORTING STATEMENT

FOR OMB CLEARANCE

Department of Health and Human Services

Administration for Children and Families

Request for Clearance for Pre-testing of Evaluation Surveys

A. Justification

1. Circumstances Making the Collection of Information Necessary

To date the Administration for Children and Families (ACF), Office of Planning Research and Evaluation (OPRE), has relied on small field tests (involving fewer than ten respondents) in developing questionnaires. However, these small tests are limited in their ability to detect and diagnose problems with the instruments and the procedures being tested. In response to this problem, we request a generic survey development clearance to allow us to use larger samples in applying methods useful for identifying questionnaire and procedural problems, suggesting solutions, and measuring the relative effectiveness of alternative solutions. Through the use of these kinds of techniques, employed routinely in the field of survey research, questionnaires can be simplified for respondents, respondent burden may be reduced, and the quality of the questionnaires used in program evaluation can be improved. An increase in the quality of the data collected through these surveys can be achieved as well. ACF has not applied for this sort of clearance before, but we have consulted with colleagues in other federal agencies, e.g., the Census Bureau, that have received generic survey development clearances to conduct extended cognitive and questionnaire design research for their surveys. ACF is requesting approval for five types of field test activities: (a) cognitive interviews, (b) respondent debriefing questionnaires, (c) split sample experiments, (d) standardized field tests, and (e) focus groups. The specific methods proposed for coverage by this clearance are described below. Also outlined are the proposed procedures for keeping OMB informed about the nature of the survey development activities being conducted.

All of the methods proposed for use in questionnaire development will be conducted among either purposive or statistically representative samples of research subjects in OPRE-sponsored research projects. Evaluation of the questionnaire and/or procedures is the main objective of the activities in this clearance. The goal of developing these surveys is to improve the evaluations of ACF programs and demonstrations. These survey respondents will include participants in ACF programs being evaluated and participants in ACF demonstrations, many of which are supported by ACF program grants, as well as comparison group members. Under this clearance a variety of surveys will be pre-tested, and the exact nature of the surveys and the samples is currently undetermined, but we expect that they will include survey activities as part of such ACF projects as the Head Start Family and Child Experiences Survey, the Innovative Strategies for Increasing Self-Sufficiency project, and the National Study of Child Care Supply and Demand, among others. The particular samples will vary based on the content of the survey being tested.

Cognitive interviews: This method involves intensive, one-on-one interviews in which the respondent is typically asked to "think aloud" as he or she answers survey questions. A number of different techniques may be involved, including asking respondents to paraphrase questions, probing questions asked to determine how respondents came up with their answers, and so on. The objective is to identify problems of ambiguity or misunderstanding or other difficulties respondents may have answering questions. This is often the first stage of revising a questionnaire.

Standardized field tests: This test serves as the vehicle for conducting standardized behavior coding of the interaction between the respondent and the interviewer. It involves applying a standardized coding scheme at the completion of a field interview, either by a coder using a tape-recording of the interview or by an observer at the time of the interview. The coding scheme is designed to identify situations that occur during the interview that reflect problems with the questionnaire. For example, if respondents frequently interrupt the interviewer before the question is completed, the question may be too long. If respondents frequently give incomplete answers, this suggests there may be some other problems with the question. An objective of standardized field tests is to collect data derived from standardized coding schemes to identify problem areas in a questionnaire in an objective and reliable manner.

Respondent debriefing questionnaire: In this method, standardized debriefing questionnaires will be administered to respondents who have participated in a field test. The debriefing form is administered at the end of the questionnaire being tested and contains questions that probe to determine how respondents interpret the questions and whether they have problems in completing the survey/questionnaire. This structured approach to debriefing enables quantitative analysis of data from a sample of respondents to learn whether respondents can answer the questions and whether they interpret them in the manner intended by the questionnaire designers. Interviewer debriefing enhances a standardized field test since it utilizes the knowledge of the survey staff that have the closest contact with respondents.

Split sample experiments: This method involves testing alternative versions of questionnaires, some of which may be designed to address problems identified in draft questionnaires or questionnaires from previous survey waves. The use of multiple questionnaires is a critical component in this type of data collection, which can include mail, telephone, or personal visit interviews or group sessions at which self-administered questionnaires are completed. Comparison of revised questionnaires against a randomly assigned control version facilitates statistical evaluation of the performance of alternative versions of the questionnaire. In any split sample experiments conducted under this clearance, alternative questionnaire versions will be tested. The number of versions tested and the number of cases per version will depend on the objectives of the test. We cannot specify with certainty a minimum panel size, although we would expect that no questionnaire versions would be administered to less than approximately forty persons or more than 250 persons in a split sample test.

Focus groups: This method involves group sessions guided by a moderator who follows a topical outline containing questions or topics focused on a particular issue, rather than adhering to a standardized questionnaire. Focus groups are useful for surfacing and exploring a range of issues that may be relevant to development and administration of a survey.

Attachment A includes examples of sections of OMB-cleared ACF surveys for which pre-testing work has been done using very small sample sizes. These surveys are examples of instrument development that would have benefited from testing with larger samples had we the time to seek OMB clearance for those activities and maintain overall evaluation project schedules. In ACF’s evaluation projects there is a need to assess the reliability of the data obtained through questions selected for the survey questionnaires from samples of respondents that represent the populations participating in the demonstration program being studied. The examples in the attachment are questionnaires that were tested with fewer than 9 respondents because there was not enough time to request OMB approval for testing with larger populations. Very often questionnaires such as these can benefit from being tested with larger samples.

To illustrate, attachment 2 contains a section of the Evaluation of Child Care Subsidy Strategies where the instrument programming is done so that all the questions are repeated for each different child care arrangement for each different child. The recall questions ask about dates when the events started and stopped and characteristics of different events. The analysts should be able to create a calendar and/or matrix with the data at the end of the testing. The testing protocol for the referenced study allowed the researchers to test the instrument with up to 9 respondents but not with a representative sample of possible respondents that varied in terms of key demographic and other characteristics of interest to the study, such as those receiving child care subsidies; those not receiving child care subsidies; those receiving TANF; those not receiving TANF; those employed; those residing in urban areas or rural areas; different families sizes; etc. The response patterns in the survey as well as level of recall required to answer these questions may have varied based on one or more of the characteristics listed, but due to limitations in the number of respondents allowed for the testing of the survey, it was not possible to gather data to test the questionnaire with all types of intended respondents and different levels of recall requirements.

In addition, the testing could have benefited from a respondent debriefing questionnaire where the investigator sits down with respondents and, with a physical calendar showing his/her responses, attempts to line up the events together with the respondents. An example of a probe to be used during the debriefing by the investigators could be the following: "I see you said that your child care arrangement started in January but I think you also said that your work started in February? Are these two dates right?" These methods, if used with enough representatives from the study’s target populations make it possible to assess: (1) if it is possible to gather the information needed to answer the study questions from diverse respondents, (2) if the data is affected by recall at different time frames after the event occurred or by number of different events to be reported on, and (3) if the method selected for data collection (e.g., by phone) yields data that meet set reliability criteria.

Procedures for Clearance

Since the types of surveys included under the umbrella of the clearance are so varied, it is impossible to specify at this point what kinds of activities would be involved in any particular test. At a minimum, however, one of the types of testing described above would be incorporated into the development testing program for each survey. With each change request, we will provide OMB with a copy of questionnaires and debriefing materials in advance of any testing activity. When split sample experiments are conducted, either in small group sessions or as part of a field test, all the questionnaires to be used will be provided. A brief description of the planned field activity will also be provided. ACF understands that OMB will make every effort to review materials for change requests received by OMB within 10 working days. All information gathered from these testing activities will be for the purpose of improving data collection instruments and procedures, not for the purpose of generating findings on the substantive topic under study. ACF will make separate submissions for clearance of full, non-developmental data collection efforts. ACF will send OMB an annual report summarizing the number of hours used, as well as the nature and results of the activities completed under this clearance.

2. Purpose and Use of the Information Collection

The information collected to develop and test questionnaires will be used by ACF to evaluate and improve the quality of the data gathered through surveys that are ultimately conducted in ACF’s research and evaluation studies. The data collected under this clearance will be published only if it is of methodological interest; it will not be presented as findings on the substantive topic under study. .

3. Use of Improved Information Technology and Burden Reduction

When the survey being pre-tested employs automated methods for its data collection, the questionnaire research conducted under this submission will also utilize automated data collection techniques.

4. Efforts to Identify Duplication and Uses of Similar Information

This research does not duplicate any other questionnaire design work being done by ACF. The purpose of this clearance is to better inform and improve the quality of ACF’s research and evaluation. Pre-testing of the scale envisioned here would not be done under other circumstances due to the time constraints of seeking clearance for each individual survey’s pre-testing plan. To the maximum extent possible, we will make use of previous information by reviewing results of previous evaluations of survey data before we attempt to revise questionnaires using additional field work sought under this clearance.

5. Impact on Small Businesses or Other Small Entities

This research to be completed under this clearance will not impact small businesses.

6. Consequences of Less Frequent Collection

This clearance involves one-time questionnaire development activities for each survey that is connected with the clearance. If this project were not carried out, the quality of the data collected in the actual surveys would suffer.

7. Special Circumstances

There are no special circumstances.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

Consultation with staff from ACF contractors carrying out research and evaluation surveys will occur in preparation for and in conjunction with the testing program for the individual surveys. These consultations will include discussions concerning potential response problems, clarity of questions and instructions, and other aspects of respondent burden. ACF has consulted with colleagues in other federal agencies, e.g., the Census Bureau, and with staff at OMB. A notice was published in the Federal Register on April 23, 2008 (vol. 73, no. 39, p. 21957), inviting public comment on our plans to submit this request. ACF received no comments or questions in response to this notice.

9. Explanation of Any Payment or Gift to Respondents

Respondents for activities conducted under this clearance may receive a small stipend depending on the types of data collection. Respondents for activities conducted in the laboratory (that is, cognitive interviews and focus groups) under this clearance will receive a small stipend.  This practice has proven necessary and effective in recruiting subjects to participate in this small-scale research, and is also employed by the other Federal cognitive laboratories.  The incentive for participation in a cognitive interview is $40, and for participation in a focus group it is $75 unless otherwise specified and approval is granted by OMB.  Respondents for methods that are generally administered as part of field test activities (that is, split sample tests, behavior coding of interviewer/respondent interaction, and respondent debriefing) will not receive payment unless there are extenuating circumstances that warrant it.

Research on incentives that may be conducted under this clearance will likely involve small nonmonetary incentives such as a flag or pen.  The Office of Management and Budget has noted that effectiveness of such incentives is a worthwhile research topic. We will work closely with OMB on incentives proposed for each research activity.

10. Assurance of Confidentiality

All respondents who participate in research under this clearance will be read a statement that will explain the study and will assure individuals of their privacy as respondents.  Each statement will be included with the change requests submitted under this generic clearance. For personal visit and telephone interviews, this information will be conveyed verbally by the interviewer. For personal visit interviews, respondents will also be notified in writing. For self-administered questionnaires, the information will be included in the mailing package, either on the questionnaire or the instructions. ACF will protect respondent data to the extent specified under the authority of the Privacy Act. An example of a typical Privacy Act statement is provided below:

ACF and its contractors will take all measures necessary to protect the privacy of individuals participating in the survey. These procedures for assuring and maintaining confidentiality will be consistent with the provisions of the Privacy Act and with ethical guidelines of professional organizations. Interviewers will attempt to conduct the interview at a time and place that allows the utmost privacy for respondents. Respondents will receive information about confidentiality protections at the outset of the interviews. They will be informed that all of the information they provide will be kept private to the extent permitted by law and that study results will be presented only in aggregate form. Participation in the survey will be voluntary. At the time of data collection for the survey, participants can choose not to participate in the survey. In addition, although every effort will be made to keep research records private, there may be times when federal or state law requires the disclosure of such records, including personal information. This is very unlikely, but if disclosure is ever required, the research team will take all steps allowable by law to protect the privacy of personal information.

11. Justification for Sensitive Questions

Most of the questions that will be included in these pre-testing activities will not be of a sensitive nature and should not pose a problem to respondents. However, it is possible that some potentially sensitive questions may be included in questionnaires that are tested under this clearance. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and alleviate them insofar as possible before the actual survey is administered.

12. Estimate of Hour Burden

The total estimated respondent burden is 3000 hours for the three year clearance period.

ANNUAL BURDEN ESTIMATES

Instrument	Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Total Annual Burden Hours
Survey development field tests, respondent debriefing questionnaires, cognitive interviews, split sample experiments, and focus groups.	1000	1	1	1000

Estimated Total Annual Burden Hours: 1000

This estimate is based on consultation with research contractors with whom we have partnered and also on advice from OMB. A variety of forms will be used in conducting the research under this clearance, and the exact number of different forms, length of each form, and number of subjects/respondents per form are unknown at this time.

13. Estimate of Cost Burden

	Number of Respondents	Responses Per Respondent	Average Burden/ Response (hours)	Total Burden (hours)	Hourly Wage Rate	Annualized Hourly Costs
Survey development	1000	1	1	1000	$17.60	$17,600

To calculate the annualized cost to respondents for the hour burden, we assume an average household income of $35,200, or 200 percent of the poverty threshold of $17,600 for a family of three. OPRE projects are expected to study low-income populations. This figure translates to an hourly rate of $17.60. The total burden to all respondents is therefore estimated to be 1,000 hours or $17,600.

14. Cost to Federal Government

We cannot anticipate the actual number of participants, length of interview,

and/or mode of data collection for the surveys to be conducted under this clearance.

Thus, it is impossible to estimate in advance the cost to the Federal Government. Costs

will be covered by the individual research and evaluation projects conducting the research from their data collection budgets.

15. Reason for Change in Burden

The change in burden is 1000 hours per year.

16. Project Schedule

This clearance is for questionnaire and procedure development purposes. Data tabulations will be used to evaluate the results of questionnaire testing. The information collected in this effort will not be the subject of any published ACF reports; however, information might be included as a methodological appendix or footnote in a report containing data from a larger data collection effort. The results of this pre-testing research may be prepared for presentation at professional meetings or publication in professional journals. Due to the nature of this clearance, there is no definite or tentative time schedule at this point. We expect work to continue more or less continuously throughout the duration of the clearance.

17. Request to Not Display Expiration Date

No exemption is requested.

18. Exceptions to the Certification

There are no exceptions to the certification.

B. Collections of Information Employing Statistical Methods

1. Universe and Respondent Selection

The data collected will be used for questionnaire development activities. For the most part, the small-scale testing activities undertaken as part of this clearance will involve purposive samples with respondents selected either to cover a broad range of demographic subgroups or to include specific characteristics related to the topic of the survey. In some instances, a probability sample may be drawn, for example, for mail surveys or to permit statistical comparisons about the effectiveness of alternative procedural treatments. A description of the plans for selecting respondents for each individual test will be provided to OMB at the time the questionnaires are submitted.

2. Procedures for Collecting Information

Data collection procedures for the testing conducted under this clearance will be varied and will most likely include personal visit, telephone, and mail surveys. Statistical results will address a variety of issues including response rates, item non-response rates, frequency distributions of data items, and analysis of behavior coding and respondent debriefing data. More specific information about data collection procedures will be contained in the description provided to OMB at the time the questionnaires are submitted.

3. Methods to Maximize Response

In general, callbacks will be used to maximize response rates in telephone field tests; reminder phone calls or letters or second questionnaires will be used to maximize response rates in mail surveys. Reminder phone calls and/or letters to participants will be used to encourage them to keep their appointments. Tallies will be kept of the number of non-respondents to all testing activities. More specific information will be contained in the description provided to OMB at the time the questionnaires are submitted.

4. Testing of Procedures

This entire submission consists of tests of data collection instruments and survey procedures. ACF expects that all the tests conducted under this clearance will result in improved questionnaires and/or procedures and thus reduced respondent burden.

5. Contacts for Statistical Aspects and Data Collection

Advice on statistical aspects of each individual survey will be sought as the testing program proceeds. Depending on the nature of the research, ACF/OPRE staff and research and evaluation contractors will have responsibility for data collection and analysis. Mark Fucello (202-401-5750) is the ACF contact person for general questions about data collection and analysis. Other contact persons for questions regarding data collection and statistical aspects of the design will be provided to OMB at the time the questionnaires are submitted.

Attachments

Examples of past surveys based on pre-tests conducted with 9 or fewer respondents.

(1) Employment Retention and Advancement Project – Section C: Employment History

(OMB NO.: 0970-0285)

(2) Evaluation of Child Care Subsidy Strategies – Sections B & C: Employment

and Child Care Arrangements (OMB No. 0970-0306)

(3) Head Start Impact Study – Parent Interview – Sections A-B: School Experience and

School Communication and Involvement (OMB No. is 0970-0229)