NSFG eLHC Feasibility and Usage 2022-Q1

NSFG OMB Attachment D3 OMB No. 0920-0314

NSFG Electronic Life History Calendar Feasibility and Usage from Year 1 Data Collection (2022)

Life history calendars (LHCs) have been used to improve recall in surveys for events that occurred in previous years or a respondent’s lifetime (e.g., Belli, Shay and Stafford, 2001). LHCs aid respondent recall through processes that capitalize on the sequential and hierarchical storage of memories and their link to landmark events in respondent’s life. Historically, LHCs have been used in interviewer administered surveys only, but with the widespread of web surveys, the need for a self-administered electronic calendar has been clear.

The NSFG has used a paper LHC in the female instrument since 1995 to facilitate recall and timing of major life events, such as educational attainment, first intercourse, cohabitation, marriage, pregnancies, and births, as well as sexual activity and contraceptive use. The collection of this life history information is key to the goals of the NSFG. While a paper form has been used as an aid in in-person interviewing in the NSFG, the introduction of a web mode of data collection starting in 2022 required the development of an electronic LHC (eLHC). Prior to the launch of data collection, the development of the NSFG eLHC was informed by data on paper calendar usage in past data collection (interviewer observations and respondent comments during interviews), cognitive lab testing of the paper calendar by the NCHS Collaborating Center for Questionnaire Design and Evaluation Research to probe how female respondents recall life events (Wilson, 2020), as well as the contractor’s small scale lab tests of the electronic calendar. The cognitive lab tests of the paper calendar suggested that while some women did not need to use it because they were able to easily recall life events, for some women it was considered essential. The small-scale lab testing of the eLHC in development prior to launch using an abbreviated version of the NSFG interview solicited feedback about the visual display, functionality, and usefulness in recalling events.

To better understand the use of a self-administered eLHC, and the life history calendar in general after the start of data collection, we embedded debriefing questions at the end of the NSFG female instrument asked of both web and in-person interviewed respondents. Because mode is confounded with the electronic calendar, this design did not allow for the direct comparison of estimates between web and face-to-face (electronic vs. paper). Evaluation of the life history calendar by mode is postponed until the mode experiment can be conducted. Thus, results presented below are informative for an initial feasibility test, with face-to-face (paper) calendar results used for reference only, not comparison. This feasibility study allows us to assess the proportion of web respondents who utilize the calendar, where it is used, and ease of use.

As has been done in past NSFG surveys, interviewers were asked to make observations on calendar use as part of the general interviewer observations form they complete while the respondent completes the CASI portion of the interview. These questions aimed to get some understanding of the extent to which respondents use the calendar, whether respondents find it helpful, whether using the calendar led them to make changes to previous responses as they recalled events better, and reasons for not using the calendar. Results from the respondent debriefing questions in the Computer Assisted Web Interviewing (CAWI) and the Computer Assisted Personal Interviewing (CAPI) modes of data collection, and from the interviewer observations in the CAPI mode are presented below. As noted above, calendar use in face-to-face administration (based on interviewer observations), is provided for reference, rather than comparison to web. In 2022, 1,165 females completed the NSFG survey in CAWI and 416 completed the survey in CAPI. Debriefing questions were presented in CAPI in Quarters 3 and 4 of 2022 and 224 female respondents provided a substantive response to at least one debriefing question¹; debriefing questions in CAWI were collected in Quarters 1 to 4 of 2022 and 1,125 female respondents provided a substantive response to at least one debriefing question². Interviewer observation debriefing questions were included in Quarters 1 to 4 of 2022 and completed for 412 of the 416 CAPI interviews in 2022.

1. Use of the Calendar

Female respondents who reported they used the calendar were asked to comment whether they found the calendar useful.

Among web respondents, around 20% found the calendar to be essential (2%), helpful (12%), or neither helpful nor unhelpful (6%). Almost 80% (873) said they did not use or look at the calendar while completing the survey.

For reference, among CAPI respondents, almost 2/3 found the calendar to be essential (26%), helpful (27%), or neither helpful nor unhelpful (10%). One third (32%) of CAPI respondents did not use the calendar (67). Another 4% found it to be not helpful.

The interviewer observations show similar results. About 2/3 of interviewers reported that the calendar was essential (18%), helpful (33%), or neither helpful nor unhelpful (17%) for the respondent. About a quarter of respondents (21%) were reported by the interviewer to have not used the calendar. Interviewers estimated that the calendar was not helpful for 10% of respondents.

2. Reasons for Not Using the Calendar

Those who reported not using the calendar were asked to select reasons for not doing so.

Among female CAWI respondents who reported not using the calendar (873), most indicated that they did not need the calendar to recall events (77%), and 21% reported not needing the calendar because they had few events to report³.

For reference, similar reasons were reported by CAPI respondents. Among CAPI respondents who did not use the calendar (67), “Didn’t need the calendar to recall events” was the most endorsed reason among CAPI respondents (selected by 65%), followed by “Didn’t need it because I had few events to report” (selected by 35%)⁴.

Interviewers were also asked to provide an opinion why respondents did not use the calendar, when they indicated the respondent did not use the calendar at all (88). The most endorsed option by interviewers was that respondents did not need it (selected for 78% of the cases).

3. Sections where Respondents Used the Calendar

Female respondents who reported using the calendar were asked in which sections of the survey they utilized the electronic LHC.

Among CAWI respondents who selected at least one response to this debriefing question (244), the most frequently selected response options were those concerning sexual activity – 53% reported using the calendar in that section, followed by contraceptive use (41%), pregnancies (37%), and marriages/cohabitation (32%).

For reference, the responses from CAPI respondents show similar results. Among those who reported at least one section of the survey instrument where they used the calendar (128), the majority of CAPI respondents selected sexual activity (72%) and contraceptive use (61%). The pregnancies and marriages/cohabitation sections were selected by more than half of the respondents who reported using the calendar for at least one section (51% and 56%, respectively).

4. Regularity of Calendar Use

Female respondents were asked to report how much they used or looked at the calendar. Among those who used the calendar in CAWI, one quarter (24%) reported using it throughout the survey. Most respondents noted that they started using the calendar, but stopped (34%), or at first did not use the calendar, but used later (26%).

For reference, among those who used the calendar in CAPI, over half (55%) reported using the calendar throughout the survey, followed by 27% who said they started using the calendar but stopped.

Interviewers reported higher estimates for respondents using the calendar throughout the survey—over two-thirds (69%)—followed by a quarter (24%) who started using the calendar but stopped.

5. Ease of Calendar Use

Female Respondents and interviewers were asked to indicate how easy or difficult it was to use the calendar.

Among CAWI respondents who used the calendar and answered that debriefing question (207), around 4/5 reported that it was easy to use (43%) or somewhat easy to use (36%); 17% of CAWI respondents found the electronic calendar somewhat difficult to use.

For reference, among CAPI respondents who used the calendar and answered how easy or difficult it was to use it (131), the majority (95%) found it easy to use (64%) or somewhat easy to use (31%). Only 5% reported it to be difficult to use.

Interviewers made similar observations – ease of use was reported for most respondents (78%) by interviewers.

6. Response Change due to Calendar

Among those who used the calendar, respondents and interviewers were asked if using the calendar prompted them to change some of their responses. Around ¾ of CAWI respondents (76%) reported they did not change their answers as a result of using the calendar.

For reference, the majority of CAPI respondents (83%) also reported they did not change their responses because of the calendar cues. Interviewers provided the same estimate (84%), indicating most of their respondents did not change their answers.

7. Summary of Findings

As described above, the results presented here are informative for an initial feasibility test of the eLHC used in CAWI NSFG interviews. Face-to-face (paper) calendar results and information from interviewer observations of CAPI interviews were also presented for reference only. This feasibility study allows us to make preliminary assessments of the proportion of web respondents who utilize the calendar, where it is used, and ease of use.

The electronic LHC does not seem to be used as frequently for CAWI respondents as the paper LHC used in the in-person interviews – the majority of respondents report not using the calendar (78% in CAWI vs. 32% in CAPI). The electronic calendar seems to be used more sporadically by respondents – more than half of the CAWI respondents (60%) reported using the calendar at some point, but not throughout the survey. In contrast, 38% of CAPI respondents reported using the calendar at some point and at least half used it throughout the survey. One factor that could influence eLHC use or non-use may be screen size – a large proportion of web respondents used mobile phones which have smaller screens on average, thus less of the calendar information can be seen without scrolling.

The electronic calendar was mostly utilized in the same sections where face-to-face respondents used the calendar - sexual activity and contraceptive use, followed by pregnancies and marriages and cohabitation.

The electronic calendar was reported to be easy to use or somewhat easy to use by respondents in the web self-administered survey.

References

Belli, R. F., W. L. Shay and F. P. Stafford (2001). "Event History Calendars and Question List Surveys: A Direct Comparison of Interviewing Methods." Public Opinion Quarterly 65(1): 45-74.

Willson, S. . (2020). Cognitive Interview Evaluation of the National Survey of Family Growth Life History Calendar . National Center for Health Statistics - CCQDER. Hyattsville, MD
https://wwwn.cdc.gov/qbank/report.aspx?1213

1 Excluding Don’t Know and Refuse responses

2 Excluding Don’t Know and Refuse responses

3 One respondent provided a Don’t Know response and was not included in analyses

4 Four respondents provided a Don’t Know/Refused response and were not included in analyses

Page 3 of 3