Pretesting Supporting Statement A

SUPPORTING STATEMENT A

FOR OMB CLEARANCE

Pre-testing of Evaluation Surveys

0970-0355

September 2011

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

A. Justification

1. Circumstances Making the Collection of Information Necessary

The Administration for Children and Families (ACF), Office of Planning Research and Evaluation (OPRE), requests a renewal of this generic survey development clearance to allow us to use samples of more than nine participants in applying methods useful for identifying questionnaire and procedural problems, suggesting solutions, and measuring the relative effectiveness of alternative solutions. Through the use of these kinds of techniques, employed routinely in the field of survey research, questionnaires can be simplified for respondents, respondent burden may be reduced, and the quality of the questionnaires used in program evaluation can be improved. An increase in the quality of the data collected through these surveys has been achieved over the past three years, warranting continued use. See Attachment B for more information.. ACF is requesting approval for seven types of activities: (a) cognitive and usability laboratory and field techniques, (b) behavior coding (c) exploratory interviews (d) respondent debriefing questionnaires, (e) split sample experiments,(f) focus groups, and (g) pilot studies/pretests. The specific methods proposed for coverage by this clearance are described below. Also outlined are the proposed procedures for keeping OMB informed about the nature of the survey development activities being conducted.

All of the methods proposed for use in questionnaire development will be use either purposive or statistically representative samples of research subjects in OPRE-sponsored research projects. Evaluation of the questionnaire and/or procedures is the main objective of the activities in this clearance. The goal of developing these surveys is to improve the evaluations of ACF programs and demonstrations. These survey respondents will typically include participants in ACF programs being evaluated; participants in ACF demonstrations, many of which are supported by ACF program grants; comparison group members; and other relevant populations, such as individuals at risk of needing ACF services. Under this clearance a variety of surveys will be pre-tested, and the exact nature of the surveys and the samples is currently undetermined. Example ACF projects that have benefited from this generic IC include survey activities as part of such ACF projects as the Head Start Family and Child Experiences Survey (FACES), and the National Survey of Early Care and Education, among others. The particular samples included in future generic ICs will vary based on the content of the survey being tested.

• Cognitive and Usability Laboratory and Field Techniques: A qualitative methodology that refers to a set of tools employed to study and identify errors that are introduced during the survey process. These techniques are generally conducted one-on-one with respondents. Cognitive techniques are generally used to understand the question-response process, whereas usability is generally used to understand the physical features of a survey, for instance, its display and navigational features. In concurrent interviews, respondents are asked to think aloud as they actually answer the survey. In retrospective interviews, respondents answer the survey as they would normally, then ‘think aloud’ afterwards. Other techniques, which are described in the literature and which will be employed as appropriate include: follow-up probing, memory cue tasks, paraphrasing, confidence rating, response latency measurements, free and dimensional sort classification tasks, and vignette classifications. The objective of all of these techniques is to aid in the development of surveys that work with respondents’ thought processes, thus reducing response error and burden. These techniques are generally very useful for studying and revising a pre-existing questionnaire. ACF has used cognitive interviews in previous generic clearance projects to revise questionnaires and has broadened the methodology request at OMB’s suggestion.

• Behavior Coding: This test serves as the vehicle for conducting standardized behavior coding of the interaction between the respondent and the interviewer. It involves applying a standardized coding scheme at the completion of a field interview, either by a coder using a tape-recording of the interview or by an observer at the time of the interview. The coding scheme is designed to identify situations that occur during the interview that reflect problems with the questionnaire. For example, if respondents frequently interrupt the interviewer before the question is completed, the question may be too long. If respondents frequently give incomplete answers, this suggests there may be some other problems with the question. An objective of standardized field tests is to collect data derived from standardized coding schemes to identify problem areas in a questionnaire in an objective and reliable manner.

• Exploratory Interview: A technique where interviews are conducted with individuals to gather information about a topical area. These may be used in the very early stages of developing a new survey. They may cover discussions related to administrative records, subject matter, definitions, etc. Exploratory interviews may also be used to investigate whether there are sufficient issues related to an existing data collection to consider a redesign.

• Respondent debriefing questionnaire: In this method, standardized debriefing questionnaires will be administered to respondents who have participated in a field test. The debriefing form is administered at the end of the questionnaire being tested and contains questions that probe to determine how respondents interpret the questions and whether they have problems in completing the survey/questionnaire. This structured approach to debriefing enables quantitative analysis of data from a sample of respondents to learn whether respondents can answer the questions and whether they interpret them in the manner intended by the questionnaire designers. Interviewer debriefing enhances a standardized field test since it utilizes the knowledge of the survey staff that have the closest contact with respondents.

• Split sample experiments: This method involves testing alternative versions of questionnaires, some of which may be designed to address problems identified in draft questionnaires or questionnaires from previous survey waves. The use of multiple questionnaires is a critical component in this type of data collection, which can include mail, telephone, or personal visit interviews or group sessions at which self-administered questionnaires are completed. Comparison of revised questionnaires against a randomly assigned control version facilitates statistical evaluation of the performance of alternative versions of the questionnaire. In any split sample experiments conducted under this clearance, alternative questionnaire versions will be tested. The number of versions tested and the number of cases per version will depend on the objectives of the test. We cannot specify with certainty a minimum panel size, although we would expect that no questionnaire versions would be administered to less than approximately forty persons or more than 100 persons in a split sample test.

• Focus groups: This method involves group sessions guided by a moderator who follows a topical outline containing questions or topics focused on a particular issue, rather than adhering to a standardized questionnaire. Focus groups are useful for surfacing and exploring a range of issues that may be relevant to development and administration of a survey.

• Pilot Studies/Pretests: These methodologies are used to test a preliminary version of the data collection instrument. Pretests are used to gather data to refine questionnaire items and scales and assess reliability or validity. Pilot studies are also used to test aspects of implementation procedures in addition to testing survey measurement issues. The sample may be purposive in nature, or limited to particular groups for whom the information is most needed. Alternatively, small samples can be selected to statistically represent at least some aspect of the survey population.

Attachment A provides an example of instruments previously approved under this generic IC.

Procedures for Clearance

Since the types of surveys included under the umbrella of the clearance are so varied, it is impossible to specify at this point what kinds of activities would be involved in any particular test. At a minimum, however, one of the types of testing described above would be incorporated into the development testing program for each generic IC. With each generic IC, we will provide OMB with a copy of questionnaires and debriefing materials in advance of any testing activity. When split sample experiments are conducted, either in small group sessions or as part of a field test, all the questionnaires to be used will be provided. A brief description of the planned field activity will also be provided. ACF understands that OMB will make every effort to review materials for generic ICs received by OMB within 10 working days. All information gathered from these testing activities will be for the purpose of improving data collection instruments and procedures, not for the purpose of generating findings on the substantive topic under study. ACF will make separate submissions for clearance of full, non-developmental data collection efforts. ACF will send OMB a report summarizing the number of hours used, as well as the nature and results of the activities completed under this clearance with subsequent generic IC renewal requests. A report summarizing the first three years of use is provided as Attachment B.

2. Purpose and Use of the Information Collection

The information collected to develop and test questionnaires will be used by ACF to evaluate and improve the quality of the data gathered through surveys that are ultimately conducted in ACF’s research and evaluation studies. The data collected under this clearance will be published only if it is of methodological interest; it will not be presented as findings on the substantive topic under study.

3. Use of Improved Information Technology and Burden Reduction

When the survey being pre-tested employs automated methods for its data collection, the questionnaire research conducted under this submission will typically also utilize automated data collection techniques.

4. Efforts to Identify Duplication and Uses of Similar Information

This research does not duplicate any other questionnaire design work being done by ACF. The purpose of this clearance is to better inform and improve the quality of ACF’s research and evaluation. Pre-testing of the scale envisioned here would not be done under other circumstances due to the time constraints of seeking clearance for each individual survey’s pre-testing plan. To the maximum extent possible, we will make use of previous information by reviewing results of previous evaluations of survey data before we attempt to revise questionnaires using additional field work sought under this clearance.

5. Impact on Small Businesses or Other Small Entities

This research to be completed under this clearance will not impact small businesses.

6. Consequences of Less Frequent Collection

This clearance involves one-time questionnaire development activities for each survey that is connected with the clearance. If this project were not carried out, the quality of the data collected in the actual surveys would suffer.

7. Special Circumstances

There are no special circumstances.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

Consultation with staff from ACF contractors carrying out research and evaluation surveys will occur in preparation for and in conjunction with the testing program for the individual surveys. These consultations will include discussions concerning potential response problems, clarity of questions and instructions, and other aspects of respondent burden. A notice was published in the Federal Register on June 10, 2011 (vol. 76, no. 112, p. 34077), inviting public comment on our plans to submit this request. ACF received no comments or questions in response to this notice.

9. Explanation of Any Payment or Gift to Respondents

Respondents for activities conducted under this clearance may receive a small stipend depending on the types of data collection. Respondents for activities conducted in the laboratory (that is, cognitive interviews and focus groups) under this clearance will receive a small stipend.  This practice has proven necessary and effective in recruiting subjects to participate in this small-scale research, and is also employed by other Federal cognitive laboratories.  The incentive for participation in a cognitive interview is up to $40, and for participation in a focus group it is up to $75 unless otherwise specifically justified.  Respondents for methods that are relatively low in burden (that is, split sample tests, behavior coding of interviewer/respondent interaction, and respondent debriefing) will not receive payment unless there are extenuating circumstances that warrant it.

10. Assurance of Confidentiality

All respondents who participate in research under this clearance will be read a statement that will explain the study and will inform individuals of the extent of their privacy as respondents.  In some cases, activities conducted under the generic clearance will not involve pledging confidentiality to participants. Such situations might include discussions with grantees, after which information is likely to be shared with others involved in the research project.

Individual statements will be included with each generic IC submitted under this clearance. When confidentiality is necessary – such as where the information being requested, about either an individual or an organization may be sensitive – the pledge of confidentiality will be made under the Privacy Act. For in-person and telephone interviews, this information will be conveyed verbally by the interviewer. For in-person interviews, respondents will also be notified in writing. For self-administered questionnaires, the information will be included in the mailing package, either on the questionnaire or the instructions. ACF will protect respondent data to the extent specified under the authority of the Privacy Act. An example of a typical Privacy Act statement is provided below:

ACF and its contractors will take all measures necessary to protect the privacy of individuals participating in the survey. These procedures for assuring and maintaining confidentiality will be consistent with the provisions of the Privacy Act and with ethical guidelines of professional organizations. Interviewers will attempt to conduct the interview at a time and place that allows the utmost privacy for respondents. Respondents will receive information about confidentiality protections at the outset of the interviews. They will be informed that all of the information they provide will be kept private to the extent permitted by law and that study results will be presented only in aggregate form. Participation in the survey will be voluntary. At the time of data collection for the survey, participants can choose not to participate in the survey. In addition, although every effort will be made to keep research records private, there may be times when federal or state law requires the disclosure of such records, including personal information. This is very unlikely, but if disclosure is ever required, the research team will take all steps allowable by law to protect the privacy of personal information.

11. Justification for Sensitive Questions

Most of the questions that will be included in these pre-testing activities will not be of a sensitive nature and should not pose a problem to respondents. However, it is possible that some potentially sensitive questions may be included in questionnaires that are tested under this clearance. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and alleviate them insofar as possible before the actual survey is administered.

12. Estimate of Hour Burden

The total estimated respondent burden is 2000 hours per year for the three year clearance period.

ANNUAL BURDEN ESTIMATES

Instrument	Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Total Annual Burden Hours
Survey development field tests, respondent debriefing questionnaires, cognitive interviews, split sample experiments, and focus groups.	4000	1	.5	2000

Estimated Total Annual Burden Hours: 2000

This estimate is based on consultation with research contractors with whom we have partnered, patterns of use during the first three years, and expected use over the next three years. A variety of forms will be used in conducting the research under this clearance, and the exact number of different forms, length of each form, and number of subjects/respondents per form are unknown at this time.

13. Estimate of Cost Burden

	Number of Respondents	Responses Per Respondent	Average Burden/ Response (hours)	Total Burden (hours)	Hourly Wage Rate	Annualized Hourly Costs
Survey development	4000	1	.5	2000	$16.64	$52,33,280

To calculate the annualized cost to respondents for the hour burden, we assume an average household income of $34,748, or 200 percent of the poverty threshold of $17,374 for a family of three¹. OPRE projects are expected to study low-income populations. This figure translates to an hourly rate of $16.64². The total burden to all respondents is therefore estimated to be 2,000 hours or $33,280.

14. Cost to Federal Government

Although we cannot anticipate the actual number of participants, length of interview,

and/or mode of data collection for the surveys to be conducted under this clearance, we estimate cost to the Federal Government based on costs incurred during the first three years of approval.

Based on costs in the first three years of approval, the annual estimate of costs to the Federal Government is $120,000. Costs will be covered by the individual research and evaluation projects conducting the research from their data collection budgets.

15. Reason for Change in Burden

This request is to renew the use of generic clearance for another 3 years. Due to experience gained over the first three years of approval, ACF hopes to make more use of this generic IC in the coming three years. Although we did not use all allotted burden in the first three years, we expect higher rates of use in the future and request the burden level to remain the same.

16. Project Schedule

This clearance is for questionnaire and procedure development purposes. Data tabulations will be used to evaluate the results of questionnaire testing. The information collected in this effort will not be the primary subject of any published ACF reports; however, information might be included as a methodological appendix or footnote in a report containing data from a larger data collection effort. The results of this pre-testing research may be prepared for presentation at professional meetings or publication in professional journals. Due to the nature of this clearance, there is no definite or tentative time schedule at this point. We expect work to continue more or less continuously throughout the duration of the clearance.

17. Request to Not Display Expiration Date

No exemption is requested.

18. Exceptions to the Certification

There are no exceptions to the certification.

1 As estimated by the US Census Bureau in “Poverty Thresholds for 2010 by Size of Family…” http://www.census.gov/hhes/www/poverty/data/incpovhlth/2010/tables.html

2 Calculated based on 2088 work hours in a year.