Various Demographic Pretesting Activities

Cognitive Testing for the Current Population Survey’s

Annual Social and Economic (ASEC)

Supplement questionnaire

DOC/ Census

Generic Clearance Letter:

1. You state that the purpose of this research is to understand underreporting and misreporting of participation in Government transfer programs, property income, and other income sources data and then you specifically focus on respondents with TANF, asset income and Social Security for part of 2010. Is there a reason why you focused on these three specific groups instead of other transfer programs, property income or other income sources data?

2. Why these nine states?

3. Please provide additional information about your methodology. In the consent form, you clearly layout what you’re going to do and how you are going to do it. I would like to see similar explanation in the generic clearance letter.

4. In the letter you say a contractor and in the cognitive interviewer materials you say Westat – Please update the language accordingly.

Census updated the Generic clearance letter appropriately.

Cognitive Interviewer materials:

1. If I understand correctly, it looks like the cognitive interviewer has to first read some questions from the paper questionnaire and then switch to the probes in the cognitive interview protocol, correct? If this is the case, wouldn’t it be easier to make one document them insert the cognitive interview protocol into the paper questionnaire? It seems like unnecessary burden at best and a source of measurement error at worst in its current form. At a minimum, you should add specific question numbers to the protocol so interviewers don’t have to figure out on the fly when to ask the probes.

It is correct that interviewers are to administer a series of ASEC questions and then switch to probing. Generally speaking, we agree it is best to have the probes inserted in the questionnaire we are testing - as you suggest, that is typically easier for the interviewers, and this especially true with respect to concurrent probing techniques. We felt this instrument was an exception. First, note that we are relying more on retrospective probing, as we felt it was important not to do any probing until all of the key ASEC questions covering potential sources of income had been asked. Also, the task of matching the probing to the relevant ASEC questions is made relatively easy by the fact that these interviews are concerned with exploring response error on a fairly limited number of income-related topics, namely food stamps and TANF, asset income, and partial-year recipients of Social Security/Disability income. Your suggestion to add question numbers to the cognitive probing is a good one, except that we are assessing the feasibility of varying the order in which the ASEC income sources are administered based on respondent characteristics. For example, elderly persons first get asked about sources that are likely to apply to them (e.g., Social Security, pension income), low-income persons first get asked about sources such as TANF and food stamps, whereas other participants will get asked the questions in the standard order. So the use of question numbers in the document containing the probes would not be very helpful. So instead, we are having the interviewers simply write the relevant page numbers in left-hand margin of the answer sheet, next to the key income sources expected to be of focus for a given interview. Finally, we should note that the relation between the probing document, the questionnaire, and the answer sheet was thoroughly covered in a 2-hour discussion session with the interviewers.

Page 1

2. The first probe on page 1 asks the respondent to forget about all the questions they were just asked and to report in their own words what their household income situation was last year. I’m not sure why you went with this approach instead of asking the respondent first to describe their household income situation and then administer the survey questions to see what they may have missed. I don’t see how asking them to forget about what they just heard helps you to understand areas where the respondent might be misreporting. The second probe would remain the same except the interviewer would be asking why they reported something in the beginning on not when ask the questions in the questionnaire.

In these interviews we are trying to develop a better understanding of precisely why certain income sources are under-reported in the ASEC. We do not think holding an open-ended discussion with respondents about their income sources prior to administering the ASEC is a good idea. This discussion would almost certainly influence how they answer the ASEC questions - the prior discussion may prompt respondents to report income sources that they would otherwise not report in response to the ASEC. Our approach of holding that discussion after administering they ASEC questions on income sources is try get respondents to recall and report sources that the ASEC questions failed to capture. To the extent that ASEC fails to capture an income source, we can then probe to determine the cause of this reporting error (e.g., misunderstanding of the relevant ASEC question, low saliency of the income source, etc.). Starting the discussion with a statement asking respondents to try and forget momentarily about the ASEC questions we have just asked them is partly to minimize the awkwardness of the inherent redundancy between the ASEC questions and this discussion, but also to help get respondents thinking about the money flowing into their households without the terms and classifications that are imposed upon it by ASEC.

Page 2

3. I would ask the first proxy probe on page 2 prior to starting the actual questionnaire as well.

We are not sure why you are suggesting that a probe about proxy-reporting should be asked prior to asking the ASEC questions, rather than after these questions. Prior to the ASEC questions, the respondent has only a vague notion of what the interview is going to cover - he/she has no idea of the rather large number of detailed income sources that ASEC will be asking about. Thus, the respondent will have little basis for providing a meaningful answer to such a probe. Also, we wonder if it may influence responding to ASEC questions in unexpected ways.

4. I like the second proxy probe but it might be more useful to ask with the questions instead of once at the end as a global question.

We can see your point - a global probe like this is generally not as useful and effective as a probe that is more specifically aimed at a given question/response. But as we stated above, we are very wary of probing within the ASEC instrument itself due to the possibility of influencing later responses. It should be noted that the probing is flexible enough such that the interviewers can target a probe like this to specific income sources, depending on circumstances that arise in the interviews. All of the interviewers on this project are experienced cognitive interviewers and understand that they should not get "locked" into administering the probes in a formal standardized manner, but rather follow up on observations made during the interview.

All Sources First, Then All Amounts

5. Why did you decided to ask respondent retrospectively whether or not they like this format? Are there other examples of cognitive testing that has used this method successfully? Usually, you want to ask a respondent about something right after they’ve experienced the format, not at the end of the testing.

In order to fully experience this new format, we cannot probe on this issue until after we’ve completed asking for all of the income amounts applicable to the respondent’s household. This essentially puts it at the end of the interview. At most, we expect that only a couple of minutes will generally elapse between administration of the ASEC and this probe. It should also be noted that the ordering of probes such as this is flexible, depending on the interviewer’s judgment and observations made during the interview.

6. Also, did you consider testing which reporting method respondents preferred by asking some respondents one way and some respondents the other way and gauging their reaction? I’m concerned that respondents don’t really have enough experience with this kind of questioning to have a response or haven’t experience the second method and will just satisfies?

We agree that this probe may yield little of value, and we are not placing a high priority on the probe itself. Our evaluation of the “All Sources First, Then All Amounts” format will actually rest much more on indications of difficulties occurring during the interview that seem to stem from it. For example, do respondents appear to have difficulty “matching” the income amount being asked back to the income source they reported earlier? We are trying to build an alternative approach that has undergone enough cognitive testing to be viable for further testing, either through more cognitive interviews or in the field.

7. This is one of your main research goals and I would think you would want to probe more about the best question order than just these two probes. Is there any research on which option is better?

There is very little research which would suggest that one approach is superior to the other. We know of one study that has examined this in the context of the Consumer Expenditure Survey, but expenditures and income likely have very different error properties. It is important to understand that our fundamental objective in the cognitive testing is to develop alternative approaches for collecting the ASEC data that will be tested within a much larger future program of research by the Census Bureau. We agree that a comparison between a source-first approach and the current ASEC interleafed approach is essential. But a larger set of responses is needed in order to assess the impact of the approach on total income reporting in aggregate, as well as within the various income components.

Stocks/ Mutual Funds

8. You mention that this section is trying to understand the sensitive nature of asking about stocks and mutual funds and yet you make no mention of this in the research goals. Please add language reflecting this goal to the Generic Clearance Letter or cut these probes.

Wording was added to the Generic Clearance Letter to address the sensitive nature of asset questions.

10. Shouldn’t you ask these probes after the mutual fund questions?

As explained above, we do not want to stop and probe frequently while administering the ASEC questions. It may not be obvious from looking at the ASEC document, but the questionnaire is actually much shorter than it appears. The vast majority of households have a rather limited number of income sources, so the majority of the questions appearing in the document get skipped. Thus, the probing that we do on stock/mutual fund investments will actually occur very soon after we’ve asked the ASEC questions on this topic – we expect it will be a matter of only a few minutes.

11. Shouldn’t these questions be contingent on whether or not the respondent has stocks or mutual funds? (Same for the SS and Disability probes)

Yes, and they are. Interviewers will only probe as appropriate.

SS and DI

12. You ask them if it easy or difficult to come up with an amount received for only part of the year but you don’t ask them how they came up with the answer, especially if they didn’t use their own records to answer the question. This may be useful with other probes and may help you get a better understanding of how they come up with their response to the questions.

Asking “how did you come up with your answer?” is a good generic probe that experienced cognitive interviewers know to ask as appropriate (note that we put a reminder on this sort of probe at the beginning of the Cognitive Probing document). But note that one will often observe how the respondent comes up with an answer as a function of the ASEC questions asking for an income amount. These typically begin by asking how best the respondent can report an amount (weekly, biweekly, monthly, etc.), and if the amount given initially is less than annual, there is a calculation involved in generating the annual amount. The respondent does not calculate this, the ASEC CATI/CAPI program does it. In the cognitive interviews, the interviewers will be equipped with hand calculators to make these calculations. Thus, a probe like “how did you come up with your answer?” will often not be applicable.