Attachment 8 - Preliminary Evaluation of the Tell us What you Think Experiment

Attachment 8—Preliminary Evaluation of the Round 10 “Tell Us What You Think” Experiment

The Round 10 interview included an experimental section of the questionnaire called “Tell Us What You Think.” The purpose of the section was to gather information about what types of questions might affect respondents’ engagement with the NLSY97. This section included a four-mode experiment. The respondents were randomly assigned to four experimental groups after first stratifying by their level of cooperativeness in previous rounds. The first mode asked personal subjective questions in a structured interview format; these questions about spending time came from the General Social Survey where they have been used to study personal priorities, as inputs into models studying household dynamics and division of labor, and to characterize personal traits. The second mode asked impersonal subjective questions also in a structured format. These questions were about social priorities and attitudes toward government and were taken from the 1987 General Social Survey. The third mode asked personal questions of a conversational nature. These questions asked respondents about their life and what they will remember about their life twenty years from now. The last mode was not asked any questions at the close of the interview.

Additional questions were asked in the interviewer remarks section and validation reinterview questionnaire to measure the respondent’s interest and engagement with the survey. In the interviewer remarks section, the interviewer was asked about the respondent’s comments concerning the questionnaire. In Round 10, about 10 percent of completed interviewers were randomly selected for telephone reinterview through a validation instrument. These respondents were asked how much they cared about the survey and how well the interview captures their lives. We are also interested in examining response patterns in Round 11 to see whether the questions asked in Round 10 improved subsequent respondent cooperativeness or data quality.

An analysis of the data from Round 10 indicate that the “Tell Us What You Think “ questions had very small effects on measures of respondent engagement or interest in the survey. Table A8-1 is a summary of the distribution of responses for the four experimental groups.

Table A8-1: Comparison of responses for the four experimental groups in R10.

(percentages)

		YTEL (mode=1)	YTEL (mode=2)	YTEL (mode=3)	Didn’t get YTEL (mode=4)
R10 YIR-050	1 positive	47.11%	45.53%	45.22%	47.50%
	2 negative	1.72%	1.77%	1.44%	1.61%
	3 mixed	8.35%	9.70%	7.69%	8.65%
	4 none	42.81%	43.00%	45.65%	42.24%
Round 10 Validation Q17	1 a lot	34.36%	33.87%	36.33%	33.53%
How much do you care about topics	2 somewhat	57.04%	54.31%	49.83%	53.89%
	3 not very much	5.15%	7.99%	7.27%	6.59%
	4 not at all	2.41%	1.28%	3.11%	3.59%
Round 10 Validation Q18	1 very well	53.61%	52.40%	47.06%	52.99%
Interview captures life	2 somewhat	39.86%	39.62%	43.94%	40.12%
	3 not very well	3.09%	4.79%	3.46%	3.89%
	4 not at all	1.72%	1.60%	2.77%	1.20%
Round 11 completes	Q1	39.97%	39.87%	37.37%	38.60%
	Q2	38.45%	37.71%	39.85%	38.27%
	Q3 (so far)	21.59%	22.43%	22.78%	23.13%
Round 11 contact attempts for completes	Mean 1-2	5.5 5.82%	5.49 7.36%	5.56 6.12%	5.42 6.07%
	3-6	66.48%	66.91%	65.62%	68.26%
	7 or more	27.70%	25.73%	28.26%	25.67%
Round 11 appointments so far	At least one	65.66%	68.11%	67.74%	66.61%
Round 11 broken appointments so far	At least one	10.03%	9.13%	9.70%	9.04%
Round 11 percentage don’t know and refused (for Round 11 completes).	Mean	2.00%	1.98%	1.87%	1.93%

Of the three modes which were asked the questions, mode 1 (personal, subjective, structured) elicited the highest percentage of positive comments in the interviewer remarks question in Round 10. Mode 2 (impersonal, subjective, structured) had the highest percentage of negative or mixed comments. This indicates that the respondents may have liked the personal questions better than the impersonal questions. Of the 1,227 cases that were validated, a higher proportion of respondents who were asked personal questions (mode 1 and 3) cared about the topics in the survey and felt that the interview captures their life very well or somewhat well.

As of December 1, 2007, about 25 percent of the Round 11 field period had elapsed, and we had reached about 70 percent of our Round 11 target of 7,400 completed cases. Our previous experiences indicate that variation in cooperativeness, response rate, and data quality comes primarily at the end of the field period, so we are not able to learn much from our Round 11 interviews to date. Nevertheless, we present some information for illustration. Based on our target for completed cases in Round 11, the completed cases have been classified as being in the first, second or third quartile of completed interviews, and we think that completing the interview early in the field period is an indicator of cooperativeness. In Round 11, modes 1 and 2 had higher proportions of early (first quartile) completes compared to mode 3. Mode 3 had the highest proportion of second quartile completes.

Less cooperative respondents are often elusive and hard to contact. Indicators of respondents’ willingness to complete the interview include completing a case in fewer contact attempts, making appointments, and keeping appointments. Mode 2 had the highest percentage of cases which have already made at least one appointment and the lowest percentage with at least one broken appointment. Of the cases completed so far, the mean number of contact attempts is similar in all four modes. Mode 2 has a larger proportion of cases that took fewer attempts compared to modes 1 and 3. Data quality in the Round 11 completed cases is measured by the percentage of refusals and don’t know responses in the data. Modes 1 and 2 have marginally higher mean item nonresponse rates than mode 3 as measured by the percentage of refusals and don’t know responses. Therefore, the respondents who were asked structured impersonal and personal questions may have been slightly more cooperative and provided slightly better quality data in Round 11, but the effects are small. While field interviewers felt that respondents liked the personal questions, there is a possibility that the personal, unstructured questions give respondents the opportunity to think more about the inadequacy of the rest of the interview.

Since interviewers felt that the unstructured personal questions were well-received by respondents, we retained the mode 3 questions and asked respondent two other types of questions concerning altruism and the importance of the survey in the “Tell Us What You Think” section of Round 11. We also plan to ask personal unstructured questions in Round12. If the final Round 11 data do not support that contention, we might reconsider the inclusion of those items in Round 12.