10 Day Letter

DEPARTMENT OF HEALTH & HUMAN SERVICES Public Health Service

Centers for Disease Control and Prevention

National Center for Health Statistics

3311 Toledo Road

Hyattsville, Maryland 20782

December 12, 2019

Margo Schwab, Ph.D.

Office of Management and Budget

725 17th Street, N.W.

Washington, DC 20503

Dear Dr. Schwab:

The NCHS Collaborating Center for Questionnaire Design and Evaluation Research (CCQDER) (OMB No. 0920-0222, Exp. Date 08/31/2021) proposes to conduct a cognitive interviewing study to evaluate the Life History Calendar (LHC) used by the Division of Vital Statistics (DVS) on the National Survey of Family Growth (NSFG).

The NSFG is a nationally representative household survey that gathers in-depth information on pregnancy, infertility, and reproductive health. This 75-minute, interviewer-administered survey also contains detailed event histories of cohabitation and marriages, contraceptive use, and recent periods of sexual activity. Because recall can be challenging with this level of detail, a hard-copy calendar covering the time periods referred to in various sections of the survey is offered as an aid to respondents. The LHC is intended to help respondents organize key personal events, which are then used as landmarks to cue memories of the dates of other events measured in the survey.

Background: Early work has shown that the use of a LHC to collect retrospective life-course information in a survey context can improve recall accuracy.¹ However, the task of completing the calendar can be fairly complicated. The NSFG previously evaluated the LHC for its usability. Debriefing questions during a field test assessed whether respondents could identify a specified timeframe on the calendar, and most (95%) could correctly do so.² However, a qualitative evaluation of the specific thought patterns that respondents demonstrate when using the LHC and the extent to which the LHC assists with recall to answer fertility-related survey questions has not been done. For this reason, CCQDER has been asked to cognitively test the functionality of the LHC in association with the relevant survey questions. Recruitment of respondents and interviewing would begin as soon as approval is received.

Plan of Study

The methodological design of the proposed study is consistent with the design of most NCHS/CCQDER cognitive interviewing studies, but with greater emphasis on recall difficulties and the extent to which the LHC assists with and improves recall.

This qualitative evaluation will study the functionality of the LHC in terms of each content area for which it is used. Interviewers will explore the manner in which the LHC shapes question interpretation, reduces recall burden, and improves reporting accuracy. The survey questions will first be administered without the use of the calendar. Once answers are obtained, interviewers will introduce the calendar and review the answers to the survey questions to determine whether it improves recall or alters the way respondents understand the questions. Follow-up probes will focus on the ways in which respondents make sense of the LHC and how their understandings of the calendar inform their approach to answering the associated survey questions. Additionally, each substantive area for which the calendar is invoked will be examined to determine whether the LHC functions differently for different topics, for example cohabitation, periods of sexual activity, and contraceptive use.

Analysis will be conducted using the constant comparative qualitative method and will focus on the constructs captured by each question, consistency of patterns across respondent groups, and the ways in which memory is (or is not) assisted by the LHC. Findings (like all CCQDER studies) will be documented in a final report and made publicly accessible on a searchable website at https://wwwn.cdc.gov/QBank.

Study Protocol: The questions to be evaluated are included as Attachment 1a&b. The testing procedure conforms to the cognitive interviewing techniques that have been described above and in CCQDER’s generic clearance package (OMB No. 0920-0222, Exp. Date 08/31/2021).

We propose to recruit up to 30 English-speaking women ages 18 to 49. Women from a variety of demographic backgrounds will be recruited; however, women with a child less than 4 years old will be prioritized, as the questions require more detail within the past three years.  Further, because qualitative sampling is based on theoretical relevance more than equal cell sizes, on-going analysis may reveal the need to recruit more from one group than others. 

Recruitment will be carried out through a combination of newspaper advertisements, flyers, and word-of-mouth. The newspaper advertisement/flyer used to recruit respondents is shown in Attachment 2 The 5 minute screener to be used to determine eligibility of individuals responding to the newspaper advertisement/flyer/word-of-mouth is shown in Attachment 3a. The 5 minute screener used to determine the eligibility of individuals from the CCQDER Respondent Database is shown in Attachment 3b. Note that wording of the template has been approved and is contained within our umbrella package. Only project specific information has been added to the document. It is anticipated that as many as 60 individuals may need to be screened in order to recruit 30 participants for the 60-minute interview (55 minute questionnaire and recall aide along with the 5 minute respondent data collection sheet).

Interviews will be conducted in the Questionnaire Design and Evaluation Research Laboratory as well as at off-site locations. All interviews conducted in both the Questionnaire Design and Evaluation Research Laboratory and off-site will be video and audio recorded to allow researchers to review the behaviors and body language of the respondents. These recordings will allow researchers to ensure the completeness and accuracy of their interview notes. Recordings will only be used by researchers from CCQDER and DVS who are working on the project. Recordings will remain under CCQDER staff control. There will be no external sharing of the recordings. DVS staff viewing/listening to recordings in the QDRL under CCQDER supervision have read and signed a non-disclosure affidavit.

Video or audio recording is required for this project except in the rare case that a study participant initially agrees to be video recorded during the telephone screening, but changes their mind. In that case, they will be asked if they agree to be audio recorded. If they decline to be audio recorded the interview will proceed without recording. In this case the interviewer will depend on their handwritten notes when conducting analysis. In addition, individuals who select “yes” for allowing the recording on the informed consent form, but “no” for retaining the recording for future research (final text before signatures on informed consent form), will be allowed to participate in the study.

NCHS government issued encrypted laptops will be used to video and audio record the interviews conducted off-site. Due to the size of the video recordings, the internal drive of the encrypted laptop is not sufficient for storage of the recordings. Recordings will be saved to an NCHS government issued encrypted flash drive. The encrypted flash drive is FIPS 140-2 compliant and approved for use by OCISO.

Extreme care will be taken with all recordings and paperwork from the interviews conducted off-site. Recordings and identifying paperwork will be stored in a secured travel case until returned to NCHS, at which point they will be transferred to the usual secured locked storage cabinets. Once the video and audio recordings are transferred to the QDRL Network, the recordings will be deleted from encrypted flash drive. Once deleted, the files are no longer available for use.

Upon completion of the project, video recordings will be stripped of the video by designated CCQDER staff and maintained only in audio format. Initial retention period of the audio recordings is 2 years after project completion. After the initial retention period, the recordings will be re-evaluated by the CCQDER Director to determine relevance, ongoing usefulness, and qualitative value for likely use in question evaluation research. If it is determined by the CCQDER Director in conjunction with CCQDER project-relevant staff that there is no valid reason to retain the recording, it will be destroyed by designated CCQDER staff. If the interview continues to be of value (defined as ongoing use by research staff, topic relevance, likely use for federal questions evaluation research), reassessment of the recording will occur again in 2 years.

After respondents have been briefed on the purpose of the study and the procedures that CCQDER routinely takes to protect human subjects, respondents will be asked to read and sign an Informed Consent (Attachment 4). Only project specific information has been added to the document. Respondents will also be asked to fill in their demographic characteristics on the Respondent Data Collection Sheet (Attachment 5). This document is contained in our umbrella package. The burden for completion of this form is captured in the interview.

The interviewer will then ask the respondent to confirm that she understands the information in the Informed Consent, and then state that we would like to record the interview. The recorder will be turned on once it is clear that the procedures are understood and agreed upon.

The interviewer will then orient the respondent to the cognitive interview with the following introduction:

[fill staff name] may have told you that we will be working on some questions that will eventually be added to national surveys. Before that happens, we like to test them out on a variety of people. The questions we are testing today are about fertility, family, and reproductive health. We are interested in your answers, but also in how you go about making them and remembering information about your life. I may also ask you questions about the questions—whether they make sense, what you think about when you hear certain words, and so on.

I will read each question to you, and I’d like you to answer as best you can. Please try to tell me what you are thinking as you figure out how to answer. Also, please tell me if:

• it was difficult to remember the information being asked,

• the question doesn’t make sense to you, or

• you could interpret it more than one way.

The more you can tell us, the more useful it will be to us as we try to develop better questions. Okay? Do you have any questions before we start? If yes, answer questions. If not, let’s get started.

The interviewer will follow the interview protocol, but will also ask emergent probes to better understand recall in relation to the question-response process. Examples of the sorts of probes that may be asked at the interviewer’s discretion include:

• How did you remember this specific piece of information?

• How difficult was it to remember?

• How sure are you about your answer?

• Can you give me an example of what you were thinking?

• In your own words, can you tell me what you think this question is asking?

After the interview, respondents will be given the thank-you letter signed by the Director of NCHS (Attachment 6), a copy of the informed consent document, and $40.

A burden table for this project is shown below:

Form Name	Number of Participants	Number of Responses/ Participant	Average hours per response	Response Burden (in hours)
Screener (recruited from newspaper/flyer)	36	1	5/60	3
Screener (recruited from CCQDER Respondent Database)	24	1	5/60	2
Questionnaire & Life History Calendar Worksheet (Recall Aide)	30	1	55/60	28
Respondent Data Collection Sheet	30	1	5/60	3
Total				36

Attachments (6)

cc:

V. Buie

J. Zirger

DHHS RCO

1 Freedman, D., Thornton, A., Camburn, D., Alwin, D., and Young-DeMarco, L. (1988). “The Life History Calendar: A Technique for Collecting Retrospective Data” Sociological Methodology, Vol. 18; pp. 37-68.

2 Plan and Operation of Cycle 6 of the National Survey of Family Growth. Series 1, Number 42.

4 | Page