Report on the Cognitive Pretest Study for the Current Population Survey's New Migration Supplement Questions, Revised Draft Jan 03, 2008

Report on the Cognitive Pretest Study for the Current Population Survey’s New Migration Supplement Questions

Anna Y. Chan

Statistical Research Division

Revised Draft

January 3, 2008

TABLE OF CONTENTS Page

Executive Summary 1

1 Introductions and Overview 3

2. Current Population Survey (CPS) Background 4

3. Methodologies and Procedures 4

   1. Background and Overview

   2. Cognitive Procedures

   3. Migration Supplement Question Series

   4. Respondents’ Demographic Characteristic

2. Results and Recommendations

   1. Basic CPS Questions 9

      1. INUSYR

   2. Year of Entry (YOE) Section 11

      1. YOECHK

      2. YOECHKYR

      3. YOEFIRSTCHK

      4. YOEFIRST

      5. YOENUM (Round 1 only)

      6. YOELAST (Round 1)

      7. TIMABDTOT (Round 1)

      8. TIMABDMTH (Round 1 only)

      9. ANYTRAV (Round 2 only)

      10. TIMABDTOT (Round 2)

      11. TIMABDYR(Round 2)

      12. TRV2MO(Round 2)

      13. OUTYRLAST(Round 2 only)

      14. YOELASTYR(Round 2 only)

      15. YOELAST(Round 2)

   3. Residence One Year Ago .21

      1. MIGSAM

      2. MIGCN

      3. MIGALL

      4. MIGM

      5. NXTSAM

      6. NXTCN

   4. Residents and Emigrants Abroad Section. 23

      1. HHEMIG (Round 1)

      2. UREMIG (Round 2 only)

      3. HHEMIG (Round 2)

      4. WHOOUT(Round 2 only)

      5. EIMGLIST

      6. EMI_INTRO (Round 2 only)

      7. EMISEX

      8. EMIAGE

      9. EMIREL

      10. EMICOB

      11. EMICIT

      12. EMICITB

      13. EMICOR

      14. EMIAF

      15. EMIGOV

      16. EMIAFDEP

      17. EMIGOVDEP

      18. EMIACT

   5. Transfers Section 27

      1. TRINTRO

      2. TRSEND

      3. TRSENDNUM

      4. TRSENDAMT

      5. TRREC

      6. TRRECNUM

      7. TRRECAMT

      8. Sensitivity and Appropriateness

3. Summary and Conclusions 33

4. Future Research 35

5. Acknowledgments 35

6. References 35

Appendices

1. Round 1 CPS Migration Supplement Paper Instrument Tested

2. Round 2 CPS Migration Supplement Paper Instrument Tested

3. Final Recommended CPS Migration Supplement Items

Executive Summary

Background

This report presents findings from a cognitive pretest study of questions designed for the newly proposed Migration Supplement for the Current Population Survey (CPS). Subject-matter experts in the Census Bureau Population Division (POP) were largely responsible for determining the content to be covered in the question series. The pretest covers four sections of the supplement: (1) Year of Entry; (2) Residence One Year Ago; (3) Emigrants Abroad; and (4) Monetary Transfer. The final questions will be fielded in August 2008.

Method

In the fall of 2007, we conducted two rounds of cognitive interviews with a total of 25 participants to pretest the new questions. We recruited participants through a variety of methods, including on-line advertisements, flyers, and word-of-mouth contact, looking for people who (a) were born outside the U.S. 50 states; or (b) are living with at least one other person who was born outside the U.S. 50 states.; or (c) had lived with someone a year ago and that person is now living outside the U.S. 50 states. The major goals for this cognitive interviewing are to test:

(1) Respondents’ comprehension of the newly proposed question; whether they can understand the question wording and its intents;

(2) The flow of the five sections, whether it is operationally feasible to implement all the proposed questions in the supplement;

(3) Whether there are any major difficulties for respondents to recall relevant information to answer the questions;

(4) Whether respondents find the questions on monetary transfer inappropriate or sensitive; and

(5) Whether the introductory statement explaining the purpose of asking the monetary transfer questions helps to reduce privacy concern.

Two rounds of testing were planned so that problems surfaced during the initial round of testing can be corrected and tested again in a second round of interviews. Seven interviewers conducted the initial round of testing with fourteen respondents and five of these interviewers also conducted the second round of testing with eleven respondents. Each round of interviews took about a month to complete.

Findings and Recommendations

During the initial round of testing, problems surfaced quickly for some items, especially those in the Year of Entry section (YOENUM, YOELAST, TIMABDOT, TIMABDMTH) which dealt with recalling the number of times a person has been outside the U.S. for 2 months or more, the most recent year he or she did such travel, and the total amount of time a person has been outside the U.S. As initially designed, these items required extensive efforts in order to elicit adequate responses from the participants. We also found that the Residents and Emigrants Abroad Section was inadequate in identifying people and handling the different skip patterns require for household and non-household members who were living in the household a year ago but were living outside the U.S. at the time of the interview. The procedures was refined and the UREMIG item was added to differentiate persons who were listed on the household roster as usual member from those who are no longer household members and providing appropriate skip patterns for these people. We encountered very little difficulty with the revised questions in a second round of interviews. However, we did have to terminate one interview shortly after it began due to language barrier despite the respondent’s ability to pass our initial screening and was considered fluent in English. Because of this issue, and given this supplement was only available in English, we have further revised some of the question wording after our second test to eliminate difficult vocabularies, problematic and ambiguous concepts. We also reduce the length of questions, whenever possible, to improve comprehension. During our debriefing sessions, several respondents have provided such sentiment. “In a survey, questions should not be too long."

Despite our original concern, the majority of our respondents who have sent money to or received money from family and friends outside the United States did not express concern or discomfort with the administration of those questions. In fact, several of the respondents commented that they are proud to report about the money they sent to family and friends because it reflects positively on their ability to support them financially. Another respondent commented how important it is to study their allocation of resources. However a couple of respondents expressed privacy concerns. They were willing to report that there are monetary transfers but were reluctant to provide the frequency and amount of transfers. They liked the current introduction, which helps broach the subject of money. However, some respondents suggested a more explicit statement that explains the purposes and uses of the survey. They believe such statement would alleviate privacy concern and legitimize the need for such data. Others commented that their initial incipient concerns were allayed by the initial confidentiality reassurance and explanations of the basic nature of the survey.

Finally, we found that it was operationally feasible to implement all the proposed questions in the supplement. However, given the monetary transfer section did raise some privacy concern for some respondents; it is recommended to keep that section at the end of the supplement so that it will not inadvertently affect the item response rate for other sections.

1. Introductions and Overview

The U.S. Census Bureau has proposed a new Migration Supplement for the August Rotation of the 2008 Current Population Survey (CPS). The Immigration Statistics Staff (ISS) of the Population Division (POP) were largely responsible for determining the content to be covered in the supplement. The supplement contains five sections and will be administered to both U.S. born and foreign-born residents with the exception of the Citizenship and Year of Entry sections, which apply only to foreign-born residents. The five sections include:

1. Citizenship

2. Year of Entry

3. Residence One Year Ago

4. Emigrants Abroad of all residents; and

5. Monetary Transfer

The aim of the supplement is to collect data that are currently unavailable to analysts. The Citizenship section will provide new data for analysts to estimate the annual naturalization rates of respondents who were non-citizens at the initial interview. The Year of Entry Section verifies the year of entry data collected in rotation 1 and collects new information about people coming to and traveling out of the U.S., which allows for better estimates on immigrant’s actual length of stay in the U.S.. The Residence One Year Ago Section collects information on where people in the household were living one year ago and allows for reliability check on the responses on last entry and comparison to estimates obtained from other surveys. The Residents and Emigrants Abroad Section provides new data on emigration and basic demographics of individuals who are currently living outside the U.S. Finally, the Transfer Section collects new data on the occurrence, frequency and total amount of household monetary transfer to family and friend who are living outside the U.S. It provides a glimpse to how immigrants maintain ties and support with people in their home country

Census Bureau staff drafted a migration supplement question series designed to capture the above information. Appendices A and B present the questions provided by ISS staff and were tested in this research. In accordance with both Census Bureau policy and with accepted best questionnaire design practices, staff of the Statistical Research Division (SRD) and Population Division (POP) conducted a cognitive interview evaluation of the proposed new question series in the Fall of 2007. This report describes that evaluation and its results.

The remainder of this report is organized as follows: Section 2 presents a very brief overview of the CPS program. Section 3 describes the research methods and procedures used in the current study. Section 4, comprising the bulk of the report, presents our findings and recommendations. Section 5 provides summary and conclusions for this study. Section 6 discusses remaining issues for future research.

2. CPS Background

The Current Population Survey (CPS) is a monthly survey of about 50,000 households conducted by the U.S. Census Bureau for the Bureau of Labor Statistics. Each month's sample is composed of eight panels that rotate on a schedule of 4 months in, 8 months out, 4 months in, so that only 25 percent of the households differ between consecutive months. The Census Bureau has conducted this survey monthly since 1942. The CPS collects the primary source of information on the labor force characteristics of the U.S. civilian non-institutional population. Respondents are interviewed to obtain information about the employment status of each member of the household 15 years of age and older. A household respondent, who must be a knowledgeable household member 15 years old or over, provides information for each household member. Some supplemental inquiries require response by a designated sample person.

Estimates obtained from the CPS include employment, unemployment, earnings, hours of work, and other indicators. Supplemental questions to produce estimates on a variety of topics including school enrollment, income, previous work experience, health, employee benefits, and work schedules are also often added to the regular CPS questionnaire. These data are collected using either the computer-assisted personal interviewing (CAPI) or the computer-assisted telephone interviewing (CATI). The CPS used addresses from the Census 2000.

3. Methodology and Procedures

3.1 Background and Overview

We used cognitive interviews to pretest the new Migration Supplement question series. Cognitive pretesting of questionnaires is a heuristics method to detect problems in one or more components of the response process and identify possible causes of survey response error. In this method, respondents are often asked to “think aloud,” to paraphrase a question’s intent, to offer their understanding of the meaning of key terms and phrases (Hughes, 2003), and otherwise asked probing questions to reveal their thought processes as they answer the survey questions. Cognitive testing generally uses small numbers of individuals (between 5 and 15) in an interview round (Willis, 2005, p7).

Between mid August and early November 2007, we conducted a total of 25 cognitive interviews in two rounds of testing. Four of the 25 interviews (two sets of two) were conducted with a mother-daughter unit from the same household by two separate interviewers simultaneously. This allowed us some ability to assess potential reporting errors. Among the 23 households, only two did not have any foreign-born residents. One of these households was hosting a foreign exchange student who has left in June this year; the other household has one household member who was studying abroad at the time of the interview. We collected data for a total of 82 individuals from 23 households. Table 1 summarized the race/ethnicity of all individuals by their born status. There are 35 individuals who were born U.S. citizens (4 were born aboard to U.S. parent(s), 1 was born in the U.S. Territories.) The remaining 47 were born outside the United States and its territories; 26 had become naturalized citizen. Sixteen of our sample was Asian, 21 was Black, 23 were non-Hispanic White, 21 were Hispanic (16 reported their race as white, 5 reported they were white), and one person is of two or more races.

Table 1

Race/Ethnicity by Citizenship of All Individuals

Citizenship Race/Ethnicity	Born U.S. Citizens (Native Born) (N=35)			Foreign Born (N=47)
	Born in the U.S.	Born in U.S Territories	Born Abroad of American Parent(s)	Naturalized Citizen	Non-Citizen	Total
Asian	3	0	0	10	3	16
Black	8	0	1	3	9	21
White non-Hispanic	10	0	3	4	6	23
White Hispanic	2	1	0	2	0	5
Other Hispanic	6	0	0	7	3	16
Two or more races	1					1
	30	1	4	26	21	82

Many interviews were conducted off-site at a location more convenient to the participant and less than half of the interviews were conducted in the Statistical Research Division’s cognitive laboratory. Participants were recruited through a variety of methods, including on-line advertisements, fliers, and word-of-mouth contact. After initial recruitment, they received a follow-up telephone call to remind them of their appointments. Since this supplement collects information on migration and emigration of U.S. population, recruiting efforts focused primarily with people who (1) were born outside the U.S. 50 states and D.C.; or (2) live with someone born outside the U.S. 50 states and D.C; or (3) lived with someone last year who are now abroad.

Seven interviewers conducted the cognitive interviews using a paper-and-pencil mock-up of what will be, in the actual CPS instrument, part of an automated (computer-assisted) questionnaire. We designed a paper version of the instrument with explicit check items written for the interviewers that an electronic instrument generally would have performed the necessary skips. Only two interviewers were experienced cognitive interviewers and most were recently trained on cognitive interviewing techniques. Hence, all interviewers participated in an one-day practice session/ refresher training where complicated examples were provided for role-play. Each was provided with a standard interview protocol and a survey instrument consisting of all new Migration question series along with scripted standardized probes and debriefing questions to maximum the uniformity and quality of our cognitive interviews. In the first round of cognitive interview, we administered the initial question series to fourteen respondents; as described in Section 4. As expected, these early interviews revealed problems with several questions almost immediately given these items were identified as potentially problematic during our initial expert review. We revised these problematic questions from the first round of interviewing and administered a second round of interviews to eleven additional respondents. In the end, we found that the most of the revised questions worked very successfully.

3.2. Cognitive Interview Procedures

While administering the new question series, we used “concurrent” methods – asking respondents to “think aloud” as they answered the migration supplement questions, and we also used retrospective probing about hesitations and other indicators of uncertainty or confusion after respondents provided a response to the question to explore the meaning of terms and phrases as they were encountered. We trained respondents to use the think-aloud technique before the interview began. After the completion of the Migration Supplement survey, we administered a set of general and specific debriefing questions asking respondents to provide feedback and comments on any issues they encountered while answering the questions, and, in particular, whether they found the questions about each of the section “strange” or “inappropriate.” We also asked respondents whether they felt they had enough information to answer the questions adequately, and whether they wondered why such questions were asked. For some of the questions that were highlighted as potentially problematic, an alternative question wording is read to the respondent asking their opinions and preferences of two different versions of the question wording.

All interviews were tape-recorded – with participants' permission – to facilitate analysis of the results. Each interview lasted about 20 to 60 minutes, depending on the number of household members that each respondent reported and the presence and nature of any difficulties each experienced. Participants were informed that their response was voluntary and that the information they provided was strictly confidential, to be seen only by Census Bureau staff directly involved in the research project. Respondents were paid $40 for their participation.

3.3. Migration Supplement Question Series

We began each interview with a standard household roster question to enumerate all current residents at the respondent’s current address. We also administered some basic CPS demographics questions, in particular, age, race, place of birth, citizenship and year of entry questions, to determine each household member’s eligibility for administering appropriate questions in the migration supplement series. We administer the Migration question series to all members of the household one section at a time prior to moving on to the next section. Consistent with the CPS design, each respondent provide self and proxy information for each member of the household. The two versions of the paper instruments used in the two rounds of cognitive study are included in the Appendices.

Only four sections of the Migration Supplement were tested in this study. The citizenship section was unnecessary for the pretesting study given the questions in the citizenship section have shown to work in the current CPS survey and that it is designed as a followup interview requiring input data from the initial CPS interview. Hence, it is inappropriate to test the followup citizenship questions in this pretest study. The four sections of the Migration supplement that were tested include: (1) Year of Entry Section which verifies the year of entry data collected in rotation 1 and collects new information about people coming to and traveling out of the U.S. to get better estimates on immigrant’s actual length of stay in the U.S.; (2) Residence One Year Ago Section collects information on where people in the household were living one year ago; (3) Residents and Emigrants Abroad; and (4) Transfer Section which collects data on the occurrence, frequency and total amount of household monetary transfer to family and friend who are living outside the U.S.

3.4 Respondents’ Demographic Characteristics

We interviewed respondents of a wide range of ages, from different race/ethnicity background, with various level of English competency (if English is a second language) and education, and hail to the U.S. from countries all over the world: Asia, Africa, Central and South Americas and Europe. Table 2 presents a demographic summary of the 25 participants. The average age of our respondents is 42, ranging from age 19 to 74. About two-thirds of our respondents are female. All respondents have completed at least a high school degree and 16 of them have completed a bachelor’s degree or post-graduate degrees. Six of our respondents are Asians, five are blacks, seven are Hispanic whites and seven are non-Hispanic whites. Only six of our respondents are U.S. citizens by birth. Nineteen of our respondents were non-citizen by birth, 13 of them had become naturalized citizen. They came from nineteen different countries such as Cuba, Ecuador, France, Ghana, India, Lithuania, the Philippines and Trinidad. All but three of these respondents speak English as a second language; for these three respondents English is the only language they speak. Two of the native born respondents are bilingual and spoke a language other than English as their first language. Our native born respondents are represented in each of the ethnic/racial group: one is Asian, two are black, one is non-Hispanic white and two are Hispanic white. The majority of our respondent reside in family households (married couple with no children in the household, married couple with child(ren), relatives households), only one respondent was in a one-person household living arrangement and another was residing in a non-relative household. The average household size is 3.5 persons.

Table 2

Demographic Summary of Cognitive Interview Participants

Demographic Characteristics	Number of Participants
Age 19-29 30-39 40-49 50-69 70 or over	5 5 7 6 2
Gender Female Male	18 7
Education Completed High school Some college College degree Graduate school	5 4 7 9
Race/Ethnicity Asian Black White Non-Hispanic White Hispanic Other/Hispanic	6 5 7 3 4
Birth Place United States (50 U.S. states & DC) U.S. Territories Born abroad to U.S. citizen parents Outside the 50 U.S. states & DC Africa Asia Europe Latin America (include Caribbean, Mexico, Central & South America)	4 1 1 19 2 5 6 6
First Language English Non-English	5 20
Household Type* Family Household Married couple household with at least one child under 18 Married couple household with adult children only (18 & over) Married couple household with no children in the household Male-headed household Non-relative households One person household House-mates	9 6 5 1 1 1
Household Size Average Range	3.5 1 to 6
*There were 25 respondents from 23 households

4. Results and Recommendations

In this section, we present cognitive results for each question in the Migration supplement series. The question texts for all questions are provided in bold and italics immediately after the item’s name. Recommended revised question wording is presented in bold only.

4.1. Basic CPS questions

We administered several basic CPS questions from the first CPS rotation to our participants. Of key interest is the INUSYR item, which collects information on when a person born outside the U.S. came to live here. The YOECHK item in the Year of Entry (YOE) Section verified the response to this item. The response will determine the skips and fills to many items in the YOE section. Because of its importance, we summarized the findings on the INUSYR item below even though this item was not part of the Migration Supplement because it has been used in the basic CPS instrument and is not subjected to the Census pretesting requirement.

4.1.1. INUSYR

When did [you/name] come to live in the United States?

(What is your best guess when [you/NAME] came to live in the United States?)

Overall, all our respondents demonstrated that they understood this basic question very well. The majority of our respondents (23 out of 25) had at least one household member who was born outside the United States and was administered this question. The response accuracy for this item is very high and took little or no time for respondents to recall the date. This is true even for those who were providing proxy information. It is apparent that the date when they first came to live in the United States is a salient and memorable even, it is not something they can easily forget. Several of them not only provided the year, they were able to report the month and sometimes even a day. Many of them also tie the time of entry with other landmark events (e.g. it’s the first time they have ever left their home country, or there was a hurricane at the time, or it’s Christmas or September 11 happened right after the respondent arrived, or the date has become a password to their daily computing activity), which according to them, makes it impossible for them to forget. However, we identified three potential issues:

1. Assumption: Although most people born outside the U.S. came here to live, not everyone who we interviewed considered himself or herself as ‘living’ here. The question makes inaccurate assumption about the person’s status in the United States. The question assumes the person we are collecting information ‘live’ in the United States. The concept ‘live’ is clear and well understood by all respondents. It is interpreted as a permanent state and is attached to the concept of home ownership and prolonged stay. The key is that “living means one came and never went back’ and that the person invests time and effort in the community, that is, they plan to ‘stay.’ We encounter one situation while administering this item where the respondent’s parents are ‘staying’ with him at the time of the interview but the respondent didn’t consider his parents as usual residents of his household when the interviewer was creating the household roster. The respondent told us that they will be staying with him six months at a time to care for his child and will return home the rest of the year, while his in-laws will come for the other 6 months. The respondent did not consider his parents ‘living’ in the United States. The interviewer listed the parents on the roster, consistent with CPS residents rule. To the respondent, this question did not apply to his parents and the interviewer found it difficult to administer this question when the respondent’s answer to the “when did they come here to live” question was “they don’t live here.’

2. Multiple responses are possible and recall is an issue. One respondent reported two responses since she has lived in the U.S. in two different time periods. Another respondent had a difficult time recalling when she first came to live in the U.S. since she had visited and stayed for months at a time multiple times before she ‘lives’ here permanently. These two European met their spouses in Europe and both have spouses in the Armed Forces. They have come to and traveled outside the United States for work as well as personal reasons. This question is likely to have response error when there are multiple entries where each was over 2 months. Note that CPS field operation instructs FR to record the latest date when multiple responses are provided; hence, it is likely that a different response will be recorded in the verification question (YOECHK) in the Migration Supplement.

3. Lack Information. Two of the respondents indicated they had to provide their best-guessed responses. One respondent was too young to remember when she entered the country but she vividly remember attending the Head Start program so she was able to place her year of entry to around or before the age of 3. The other respondent was reporting for her parents who came to the U.S. before she was born. But she has adequate information and her responses were correct when we match them to her mother’s report (her mother was also interviewed for this project.) We decided in these cases, if the respondent is a close family member or have some kind of landmark to estimate the person’s date of entry, their responses could be very accurate. Although we did not observe problem in our interview, this highlight this can be a potential issue especially when the respondent is providing proxy information for someone in the household who is not a close family member or friend or when the year of entry occur when a respondent was very young.

Judging from the above issues, if there is any discrepancy between responses to INUSYR to the later YOECHK item, they are likely to be:

1. Correction made by the same or proxy respondent after the initial interview where the person find out either from parents or someone older his or her actual start date for living in the U.S. especially when the person was too young at the time;

2. Respondents who have multiple entry dates and even though they may have provided an earlier date of entry, the interviewers recorded the later date as instructed by current CPS field practice when multiple responses are provided, so an earlier date reported in YOECHK for these respondents, especially when spouses are in the U.S. Armed Forces may imply there were multiple periods of living in the United States;

3. Respondents who have recall issue due to multiple entries and actually may have check their records to find out the correct date and corrected an earlier rotation’s response

Recommendation for INUSYR: There is a need to revisit the question wording for the basic CPS INUSYR item and the current field instructions for multiple responses. Simple recall failure is a problem for which we have no recommended solution.

Suggested revision for the basic CPS INUSYR item:

“When is the first time [You/NAME] come to the U.S. for [2 months or more]^*?”

*Analysts provided this length of time as a definition of ‘living in the U.S. This question wording avoids using problematic terms such as ‘live’ or ‘stay’. Specifying the length of time worked well in this pretest.

4.2. Year of Entry (YOE) Section

This is the first section of the Migration Supplement tested in this study. Due to its content, this section requires the most amounts of recall and estimation from respondents. However, judging from the comparison of the self and proxy responses between the two sets of mother-daughter respondents, we believe our respondents in general are rather accurate when providing proxy responses for household members, at least those with close relationship. During the initial round of testing, problems surfaced quickly for several items in this section, in particular, YOENUM, YOELAST and TIMABDTOT. As initially designed, these items required extensive effort in order to elicit adequate responses from the participants prompting some major revision to these items for the second round of testing. The first four items of this section (YOECHK, YOECHKYR, YOEFIRSTCHK, YOEFIRST) were tested with the same question wordings in both rounds and their findings will be presented together. The results for the redesigned items, however, will be reported separately for each round.

1. YOECHK

Previously, you stated that [you/NAME] came to live in the United States in [INUSYR]. Is this the correct year?

All 23 eligible respondents who were administered this question clearly understood the question. But due to its proximity to the INUSYR, an unnatural ordering of question only due to this testing, all but one respondent reassure us that their earlier responses were correct. The sole respondent (KT-2) who elaborated on her ‘yeah’ response by mentioning that ‘but the first time I came to live here was in 1982, not 2000’. The respondent has lived in this country in two different times and had in fact provided two responses when the INUSYR item was administered earlier. Since the interviewer recorded the later date (consistent with the CPS practice, which instruct interviewers to record the latest date when multiple responses are provided), the respondent reiterate in her response to this item that she and her child first came to live here at the earlier year that she has reported in INUSYR but was not recorded.

Final recommendation for YOECHK: Although there is no indication that this question is difficult to understand, the question wording is subjected to the same issue as discussed for INUSYR. Should the question wording for INUSYR remains the same, then, the question wording for this item should also remain the same. Although it will be subjected to the assumption issue using the concept of ‘live’, the issue is minor given the majority of people we interviewed consider themselves as living here; and at a minimum, this question wording utilizes dependent interviewing, which will help reduce respondent burden and reporting error.

Previously, we recorded that [you/NAME] came to live in the United States in [INUSYR]. Is this the correct year?

2. YOECHKYR

When did [you/NAME] come to live in the United States?

As noted above, one respondent was administered this followup question for YOECHK because she and her daughter had an earlier entry, which the interviewer did not record. No issue was observed for this question.

Final Recommendation for YOECHKYR: No change is recommended but the question wording should be revised depending on the future decision on the question wording of INUSYR.

3. YOEFIRSTCHK

Did [you/NAME] EVER stay for at least 2 months consecutively in the U.S. BEFORE [you/he/she] came to live here in [YOECHKYR/ *INUSYR]?

During round 1 of our cognitive interviews, all respondent understood the question wording well. Only one out of the 13 respondents provided a ‘yes’ response to this question. This respondent’s spouse, who was a child and lived in Africa at the time, came for a yearly visits during the summer. Respondent provided a rough estimate for her spouse first visit.

During round 2 of the cognitive interviews, three respondents (AC-4, CL-4, KT-2) provided a ‘yes’ response. This question worked as intended for these respondents. In the first case (AC-4), respondent reported an earlier date for his spouse because his wife has been in the U.S. prior to living here permanently. In the second case (CL-4), respondent’s parents come to visit every year for a few months each time. As noted earlier in the discussion of the INUSYR item, this respondent did not consider his parents as living with him but the interviewer listed the parents on the roster. So when INUSYR was asked, the respondent did not think the question applies since he didn’t consider his parents as ‘living’ here. The interviewer recorded 2007, the very last time his parents came to the U.S. When YOEFIRSTCHK is administered, the respondent provided all the years that his parents have come with 2002 being the first time. Despite the assumption issue, this question works as intended and captured the earliest date of entry for these people. The third respondent (KT-2) knew she came to live in 1982 and again in 2001. The interviewer recorded the later date. Again, this question collected the earlier date as intended. Although this question seems to work as intended, we did observe two issues with this question.

1. Assumption and misinterpretation of question. One respondent who is fluent in English misunderstood the question (AC-6) because the question is asking about experiences that she did not have. “…You ask me when did I come to the U.S. that means you are asking for the first time I came here…it’s the starting point…Why did they ask if we were here before?” Given the first time respondent came to the U.S. is also the first time she came to live in this country, (this is common among most of the respondent), this question didn’t make sense to her. So at first, she interpreted the question to mean whether she came to the U.S. exactly two months before she came here to live. It was difficult for her to comprehend the question. The interviewer has to read the question three times before she understood it.

2. Recall issue. One respondent (CS-3) realized during debriefing session that she has forgotten about an earlier time that she has come to the U.S. Hence, the verification does not work when there is recall issue.

Final Recommendation for YOEFIRSTCHK: shorten final question, focus on the time frame and avoid ambiguous terms like stay and live.

Before [INYSYR], did [you/NAME] ever come to the U.S. for 2 months or more?

4. YOEFIRST

R1. What was the FIRST year that [you/NAME] came to the U.S. and stayed for

at least 2 months?

R2. What was the year that [you/name] FIRST came to the U.S. and stayed for at least 2 months?

Five respondents were administered either form of this question. All understood the question well and were able to provide a reasonable answer. However, there is less certainty in their responses, two out of five qualified their answers that it is an approximation. There is potential response error as a result of:

1. Recall/Estimation Difficulty: As one respondent providing the answer told us, she knows her response is close but it is difficult for her to recall event that took place so long ago (30 years), in particular, she has been to the U.S. several times before ‘living’ here permanently. She needs to recall all those visit and decide how long each visit was.

2. Proxy information: Two respondents provided responses based on pure estimation because they were providing proxy information for their spouse, again for events that have taken place long time ago, in one case, almost 40 years ago.

Final Recommendation for YOEFIRST

What year did [you/NAME] FIRST come to the U.S. for 2 months or more?

IF THERE IS MULTIPLE STAY LONGER THAN 2 MONTHS, ENTER THE FIRST TIME THE PERSON CAME FOR 2 MONTHS OR MORE.

5. YOENUM (Round 1 only)

Since [*YOEFIRST/**YOECHKYR/INUSYR], about how many times [have/has] [you/NAME] been out of the U.S. for at least 2 months? (Your best guess is fine).

All respondents who were administered the questions understood the question very well. Six respondents have reported having at least one of their household members who has traveled outside the U.S. for more than 2 months since the person first came to the U.S. Two respondents provided self-response and were confident with their answers. The remaining four provided proxy information either for spouse or parents. Among these respondents, we observed the following issues and problems:

1. Lack Information: In general, respondents who were providing proxy information knew that there were travels of this sort but it was difficult for these respondents to provide a precise answer to this question. For example, one respondent clearly knew that his mother travels a lot but has no idea how many times she has been outside the country for more than 2 months. “I’m just guessing, she goes to India rather frequently.” (Case KH-2)

2. Recall Difficulty: One respondent has difficulty recalling the information, especially when household members whom the respondents were providing information for were frequent travelers or they travel for work. For instance, after one respondent heard the question for her father who travels for his work, her only response was, “Wow, many many…” (Case AC-1)

3. Overly complex. Respondents are unable to remember or synthesize the details and the qualifiers in the question because of the length of the question. Some respondents were over consumed with the earlier part of the question, they didn’t ‘hear’ the ‘at least 2 months’ part and many include only personal travel and left out all work travel. Moreover, for respondents who have never traveled since they came here, this question is overly complex.

We noticed the responses to this question are subject to error because of estimation difficulty and or lack of (correct) information.

Final recommendation for YOENUM: The current question written as such requires precise responses but was overly complex for some respondents to synthesize. Given the original intent of the question is mainly for proper skips of later questions rather than obtaining data for analytical needs, we recommend replacing this item with a set of simpler questions, ANYTRAV and TRV2MO, where the first item efficiently skip persons who haven’t travel outside the U.S. to the end of the section while administering the followup item to those who have traveled to find out if any of the travel was longer than two months. And if so, they were then administered the YOELAST question.

Recommended new items for round 2 testing: ANYTRAV & TRV2MO

1. ANYTRAV (Replace YOENUM in Round 2)

Have you ever lived or traveled outside the United States for work or any personal visits since you came here in [YOEFIRST/YOECHKYR/INUSYR]? Please count all work-related or personal travel of any length.

2. TRV2MO (Placement: after TIMABYR in Round 2)

Since [First entry], were any of your trips outside the U.S. longer than two months?

6. YOELAST (R1 ONLY)

Now think of the MOST RECENT time [you/NAME] [were/was] outside of the U.S. for at least 2 months. What year did [you/he/she] MOST RECENTLY come back to the U.S.?

Six respondents were administered this question in round 1. All but one respondent understood the question well. We found two issues with this item.

1. Overly complex. One respondent misunderstood the question because he did not relate the first statement with the question. During the debriefing session, this respondent told the interviewer that this question is “tricky”, he did not understand that the question refers to the one trip when he was outside the U.S. for two months. He thought the question was asking him the very last time he was outside the U.S. only since the question did not specify the actual duration and he didn’t see the connection between the first sentence and the question. Hence in his response, he told the interviewer the last time he enter the U.S. which was not a trip that he has spent more than 2 months outside the U.S.

2. Difficult cognitive task and inadequate response option. During debriefing session we asked these 6 respondents whether they found this question easy to respond to. All but one respondent reported that the question was easy to respond. In this case, the respondent’s father frequently travels outside the U.S. months at a time for his work; it was difficult for her to come up with an answer. Also, for this person who is currently outside the U.S., this is not an appropriate question. In our recommendation for round 2, we recommended separating the questions from when a person left and when the person plan to return to the U.S.

During the respondent debriefing session, we presented an alternative version of this question to respondents with a ‘yes’ response to YOULAST as follows:

“Now think of the MOST RECENT time [you/NAME] [were/was] outside of the U.S. for at least 2 months. Approximately what year did [you/he/she] come back to the U.S. from that trip?”

Four of the six respondents prefer the revised version finding it more specific and easier to understand; the one respondent who clearly misunderstood the first version of the YOELAST understood the alternate version and provided the year he was away for two months. However, two respondents found the revised question more confusing. We decided to revise the alternate version into a three-part question for a second round of cognitive pretest to reduce the complexity of the question.

Recommended revision for round 2 testing:

A key factor in respondents’ difficulty with YOELAST as originally designed seemed to be the length of the question and respondents’ inability to connect the first statement with the question. We recommended the following question series to allow for an appropriate response for person who is still traveling outside the U.S. (YOELASTRY & YOELAST) and to simplify the response tasks for interviewee for the round 2 cognitive testing.

(1) OUTYRLAST (see 4.2.13)

Now think of the MOST RECENT time [you/NAME] [were/was] OUTSIDE of the U.S. for at least 2 months. What year did [you/he/she] leave the U.S. for that trip?

(2) YOELASTYR (see 4.2.14)

Did [you/NAME] come back to the United States that same year? [If no, ask next, else end]

(3) YOELAST (see 4.2.15)

What year [did/will] [you/NAME] return to the United States from [that/current] trip? (Your best guess is fine.)

7. TIMABDTOT (Round 1)

Including ALL trips of any duration, about how much time IN TOTAL [have/has] [you/NAME] been OUTSIDE of the U.S. (50 states and Washington, D.C.) since [year first came to the U.S.]?

This is one of the most difficult items in the entire YOE section for respondents to comprehend and to arrive at an answer. During the first round of cognitive interview, of five respondents thought it is easy for them to provide self-response answer to this question, but two thought it was difficult to provide proxy information. The question requires extensive probing in order to elicit an appropriate response. We have identified a number of problems with this question.

1. Recall and Estimation Difficulty. The response to this item is subject to error because of all the recall, calculation or estimation of the information needed in order to provide an answer. Many respondents find it difficult to recall all their travels (many admitted that they have accidentally excluded business travel and countries such as Canada) and or given they have been in this country for a long time, it is a huge cognitive task to calculate for a response for this question. One respondent provides us a clue as to why Canada was forgotten as a place outside the U.S. He reasoned that since he drove to Canada, it didn’t feel like he was outside the U.S. We suspect this will be the case with Mexico, the other country that borders the U.S. that allows for land transportation.

2. Overly complex. Many respondents misunderstood the question, some told interviewers the number of times they have been outside the United States. Others enumerated all the trips, talked about the duration of each trip but didn’t know that they have to add them all up.

3. Questionnaire effects: For some respondents, their responses to this question were impacted by the context of the previous questions: YOENUM and YOELAST. These questions were both focusing on a 2-month time frame where as this question asks about all trips of any duration. As noted above, since YOENUM asks about the number of trips, respondents provided the same count response to this item.

4. Uncertainty – although many answers were given precisely (e.g. 4 months and 1 week), and with confidence, many others came with qualifiers (e.g. “I do not know, maybe 3 months, 4 months”, “I think its 6 or 7 months’, ‘I am gonna give a ball park and say about, 10 months, maybe 11 months”). It took some respondents minutes before they were able to calculate the total time, and as noted above, it is often provided in a range or the unit of response did not match the ones that have been provided.

5. Ambiguous concepts subject to respondent interpretaion. Instructions or definitions are missing or were inadequate; respondents weren’t sure what counts as a trip.

6. Problematic Terms. The term ‘trips’ were problematic for one respondent. “It is not a trip outside the U.S.” Respondent’s spouse live in a foreign country at the time, to her, he is not making a trip outside the U.S., he was returning to his country of origin. To this respondent, the term trip is used inappropriately.

During our debriefing session, we provided an alternate version of TIMABDTOT (see below) and probed respondents their understanding of the question and their preference between the original and alternate version of the question.

Nowadays, people sometimes travel outside the U.S. for business, vacation and/or personal visits to family and friends abroad. Including all these trips of any duration, please estimate about how much time IN TOTAL [you/NAME] [have/has] been OUTSIDE of the U.S. since [YOEFIRST/*YOECHKYR/**INUSYR]. Would you say it is less than a year, about one year or more than a year?

(i) less than a year

(ii) about one year

(iii) more than a year

Twelve respondents were administered the alternate version. All but two of the respondents preferred the alternate version. They thought that the alternate version was easier to understand, more explicit and the introduction helps explain the reason why the question was asked, and the close-ended response option suggest a less than precise response is sufficient making the response task much easier for respondents. For the two respondents who prefer the original version, they thought that the revised question was too long and complex for them to comprehend. Hence, they prefer the first version, which they were able to understand better. As a result, we revised the original question and simplify the cognitive task by providing close-ended response set for the round 2 cognitive pretest. See Section 4.2.10.

8. TIMABDMTH (Round 2)

How many months would that be?

This question was used as a followup question for the TIMABDTOT question if response for the total time outside the U.S. was less than one year. We observe one major issue with this item.

1. Inadequate response option. At least six of the responses were provided in weeks and sever other responses were provided in months but with uncertainty, such as 10 to 11 months.

Final recommendation for TIMABDMTH. Given the recommended revision for TIMABDTOT, this item will be deleted and will be replaced by TIMABDYR. See section 4.2.11.

9. ANYTRAV (Round 2 only)

Just to make sure, [have/has] [you/NAME] ever lived or traveled outside the United States for work or any personal visits since [you/he/she] came here [YOEFIRST/YOECHKYR/INUSYR]? Please count trips of any duration.

This new question is administered in the second phase of our cognitive interview. Ten respondents were administered this question. Among the 10 households that have foreign-born household members, 9 have at least one member who has traveled since they came here. Of the 24 persons who were born outside the U.S. in these nine households, 18 (75%) had been outside the U.S, five have never been outside the U. S. since they came; one person has missing data on INUSYR, hence this question was skipped.

Overall, this question was easy to understand and we did not identify any issue with this revised question. Based on respondents’ answers and paraphrasing of the question, all respondents appear to understand this question very well. The goal of this question is to illicit respondents’ memory on any travel before collecting the total amount of time they spent outside the U.S., the question works as intended. However, one respondent did forget to include his trip to Canada but since he has done most of his travel back home, it seems fine to forget Canada because that was a ‘very short trip.’ We did notice if Canada is the only place a person travels to, he or she tends not to forget.

10. TIMABDTOT (Round 2 )

Now adding up ALL those TIMES [you/NAME] spent outside the U.S. since [YOE], would you say [you/NAME] spent more or less than one year outside the U.S.? (Your best guess is fine.)

Similar to the question wording tested in round 1, we observed uncertainty and answers are often provided in a range (e.g. 10 to 15 years). This is not a major problem, however, since, as indicated by the parenthetical statement – “Your best guess is fine”, and there is no need for great precision. As discussed earlier, our final recommendation for this question is to further reduce the length of this time for ease of comprehension.

Final recommendation for TIMABDTOT:

Now adding up ALL those TIMES [you/NAME] spent outside the U.S. since [YOE], would you say it is more or less than one year? (Your best guess is fine.)

11. TIMABDYR ((Round 2)

About how many years IN TOTAL [have/has] [you/NAME] spent outside the U.S.? (Your best guess is fine.)

No major issue was observed for this question other than the occasional responses that were provided in a range, as discussed above it was not an issue for the purpose of this item other than recall error where one respondent reported 4 to 5 years but later found out it was closer to 15 years when the interviewer helped her add up all her travels during the debriefing section. This is an issue for which we have no recommended solution. For the final version of this question, we recommended adding the reference period at the beginning of the question to highlight the importance of that context.

Final recommendation for TIMABDYR

Since [YOEFIRST], about how many years IN TOTAL [have/has] [you/NAME] spent outside the U.S.? (Your best guess is fine.)

12. TRV2MO (Round 2)

Since [First entry], were any of [your/NAME’s] trip outside the U.S. longer than two months?

No major issue was observed for this question no change is recommended. The change in timeframe designed for this version works as intended; no respondent was confused by the natural change in time frame in this round.

13. OUTYRLAST (Round 2 only)

Now think of the MOST RECENT time [you/NAME] [were/was] OUTSIDE of the U.S. for at least 2 months. What year did [you/he/she] leave the U.S. for that trip?

No major issue was observed for this item. However, we noticed in majority of the cases, the person traveled and came back to the U.S. within the same year making it somewhat redundant to ask two separate questions for these respondents. To reduce such burden, we decided to delete the second item (see YOELASRYR below) but modify the OUTYRLAST item.

Final recommendation for OUTYRLAST

Now think of the MOST RECENT time [you/NAME] [were/was] OUTSIDE of the U.S. for 2 months or more. What year was that? (Your best guess is fine.)

14. YOELASTYR ((Round 2 only)

Did [you/NAME] come back to the United States that same year?

No major issue was observed for this item but we recommend this item to be deleted (see above).

15. YOELAST (Round 2)

What year [did/will] [you/NAME] return to the United States from [that/current trip]? (Your best guess is fine.)

This question was rarely administered given most respondent travel outside and come back to the U.S. within the same year. We observed a couple of issues with this item.

1. Ambiguous concept subject to respondent interpretation: We observed in once case where the concept of ‘return’ was misunderstood because the term was ambiguous. This respondent had left the U.S. as her spouse was stationed outside the U.S. During that tour, they had return for a visit. Hence, she reported the year that she and her family ‘stopped by’ the U.S. while they were still living outside the U.S.

2. Interviewer difficulty/inadequate response option. The current question did not indicate how interviewer should handle cases where the person being interviewed is still traveling outside the U.S.

Final recommendation for YOELAST

What year did [you/NAME] come back? (Your best guess is fine.)

IF PERSON IS CURRENTLY OUTSIDE THE U.S. ENTER ‘9996’, ENTER LINE NUMBER FOR WHOOUT.

2. Residence One Year Ago

The findings of this section show that overall the items designed for this section work as intended. This is no major issue observe for this section. In general, our respondents were able to understand most of the questions and provided adequate and reasonable responses. We identified one problematic question, MIGALL, and provided a suggested revision for its question wording. The detail results summarized for each of the question are presented below.

1. REA_INTRO

R1. The next part of the survey is about people who are NOW living outside the United States but were living here with you last year, that is, around [TODAY’s month & day], 2006. By outside the U.S., we mean outside the 50 states and Washington, D.C.. The picture would be incomplete if we didn’t get a little information about these people you used to live with.

During our round one cognitive interview, one respondent was unable to paraphrase the introduction. Later on, the same respondent gave the same response to the introduction to the Transfers section (TINTRO). Basically, the introduction was too long for her to understand, and hence, we shortened the introduction in the second round and made some of the text optional (See below).

R2. The next part of the survey is about people who are NOW living outside the United States but were living with you last year, that is, around [TODAY’s month & day], 2006.

READ IF NECESSARY. By outside the U.S., we mean outside the 50 states and Washington, D.C.

During our second round of interview, the revised introduction seemed to work fine. But we caution that it may still be too long for CPS respondents if they have very low level of English proficiency.

2. MIGSAM

Was [Reference Person] living in this house (or apartment) one year ago?

All but two reference persons in the 23 households did not live in their current residence one year ago. One respondent was living somewhere else in the U.S., the other was living outside the U.S. with her entire family one year ago. We observe no issue with this question.

Recommendation: no change to this question

3. MIGCN

What country did [Reference Person] live in one year ago?

Of the two households where the reference person was living elsewhere one year ago, one respondent was living in another house in the U.S. and the other was living in the Netherlands Antilles with her family where her spouse was stationed there by the Armed Forces. No difficulty was observed with this question.

Recommendation: no change to this question

4. MIGALL

There are [# ] other persons in this household ages 1 year or over. Did ALL of these persons live with [Reference Person’s name] in [*this house/that house/name of country] one year ago?

This question was administered to all but one respondent who lived alone. Most households have rather stable living arrangement where all household members were living with the reference person one year ago. Two respondents provided a ‘don’t know’ response because of the living arrangement of their family and another two respondents provided a ‘no’ response to the question. We identified several issues with this question.

1. Double barrel – This refers to question that asks about two different concepts but assumes a single answer: (a) verifying the number of people over the age of 1, and (b) whether they all live with the household respondent.

2. Complex living situation. In one incidence, a spouse of one respondent was living outside the country for work one year ago. The respondent was unable to answer the question and provided a ‘don’t know’ response because the she couldn’t decide whether or not she should count him as ‘living’ with her. He technically lived there but was physically absent one year ago in August.

3. Recall – We observed that it was difficult for one respondent to recall whether her children was living with her one year ago in August given her children live at home during the summer but return to the dormitory sometime during August.

4. Ambiguous concept subject to respondent interpretation. We noticed that the phrase “one year ago” was misinterpreted by one of our respondent who took the question to mean that the person must have been “living with her the entire year.”

Final recommendation for MIGALL:

“Did [the other person/ALL other persons] in this household who [is/are] age 1 year or over [lives/live] with [Reference person] in [*this house/that house/name of country] on about August 12, 2006?”

5. MIGM

Which one of the other members of this household did NOT live with [Reference Person] one year ago?

This question is administered to household where some of the household members (age 1 or over) did not live with the reference person one year ago. We did not observe any issue with this question.

Recommendation: no change to this question

6. NXTSAM

[.../*Just to make sure] [Were/Was] [you/NAME] living in this house (or apartment) one year ago?

This question is administered to household members who did not live with reference person one year ago. Two respondents were administered this question. One respondent reported that her husband was traveling for work outside the U.S.; another respondent reported that her nephew was in her home country and wasn’t living with her one year ago. There is no apparent difficulty for any of these respondents reporting to this question.

Final recommendation: no change to this question

7. NXTCN

What country did [NAME] live in one year ago?

This question was very clear and both respondents who were administered this question reported an adequate answer. The spouse of the first respondent was in Nepal and the nephew of the second respondent was in Guatemala.

Final recommendation: no change to this question

2. Residents and Emigrants Abroad Section

Nine respondents from eight households reported someone who was outside the U.S. at the time of our interview. Our overall findings from this section suggest that the questions designed to collect the basic demographics of people outside the U.S. worked well. Only a couple of minor adjustments were made after the first round of cognitive interview and the revised questionnaire worked as intended. We refined the procedures for identifying person who used to live in the household and those who were usual household members at the time of the interview by adding the UREMIG item and revising the wording for the initial HHEMIG question. We also restricted the universe for the Armed Forces question by setting an age limit reduce respondent burden. We also recommended a minor revision to the EMIREL question.

1. HHEMIG (Round 1 only)

Is there anyone who BOTH lived here with you one year ago AND is now living outside the U.S.? Please include people in the Armed Forces as well as anyone who stayed here for at least 2 months.

Four out of fourteen respondents (from three households) in the first round of cognitive interview reported that someone was living at their household one year ago but was living outside the U.S. at the time of the interview. Of the four respondents, two were mother and daughter pair where they were reporting the absence of her spouse/father respectively. Another respondent reported her spouse being in Asia (but her daughter who was also a respondent; did not, see item 2 below). The fourth respondent reported her roommate being absent during our debriefing session (given her roommate wasn’t living with her one year ago, she correctly excluded her in this question). We observed several issues with this question.

1. Interviewer Difficulty: the series was clearly not designed to capture current household members. When respondent identified a current household member as being outside the U.S., our interviewers found it redundant to ask all the demographics question again, another thought the HHEMIG question wasn’t relevant for current household members who are temporarily away.

2. Complex living arrangement: One respondent reported that her spouse was working in Asia at the time of the interview. The interviewer probed to confirm that respondent spouse was living with her in this house one year before. The respondent responded in Chinese: “This is difficult to say. He comes home every month, his work is in New Jersey (one year ago), all his belongings, car registration and all mails come here (MD)...when he is busy we go visit him (in NJ). I am not sure he lives in this house. I don’t know whether I should say he was living with me then.” [Interviewer: Would your husband consider himself living here or in NJ?]. “He considers MD home not NJ. Also when there are calls asking for him here, I tell people he is out of town.” Interestingly enough, respondent’s daughter was also interviewed but she did not list her father on the original roster nor did she responded ‘yes’ to HHEMIG. Since she did not consider her father as living with her a year ago (“He lives in New Jersey”), this question did not apply to her.

3. Assumption. Again the term ‘lived’ assumes people define themselves as living in the U.S. rather than ‘staying’ or ‘visiting’. One respondent told us as the end of the interview that she thinks maybe she should have listed her mother as a person who was ‘living’ with her last yesterday but is now back in her home country. Although respondent’s said her mother didn’t technically ‘live’ in her house, she did visit and stay for 3 months during the reference period that this question elicits. The same situation happens for respondent’s aunt.

Recommendation for Round 2 Testing

Separate items are needed for identifying persons living outside the U.S. who are still a usual household member versus those who are no longer a household member in order to illicit the appropriate persons and to administer appropriate questions for the persons. In the second round, we recommended adding the UREMIG item (see below) to identify current household member who are living outside the U.S. at the time of the interview and revise the current HHEMIG item to exclusively identify ‘current’ non-household members.

2. UREMIG (Round 2 only)

Is there anyone who is listed as living here but is currently living outside the U.S.? Include Armed Forces members.

During the second round of cognitive interviewing, we added this question to separately collect names of people who are usual household members. Two of the eleven respondents during this round of interview reported household members who were outside the U.S. at the time of the interview. This question works as intended and no problem was observed for this item. However, we observed one issue.

Assumption. The revised question assumes respondent know whom we are referring to when we use the phrase ‘who is listed’. To avoid such confusion, the revised question will list all names of all members who are listed on the household roster.

Final Recommendation for UREMIG

This survey listed [list all roster names] as members of this household. Is any one of them living outside the U.S. now?

3. HHEMIG (Round 2)

Other than the people who are living here now, did anyone else live here with you one year ago but is now living outside the U.S.? READ IF NECESSARY: included Armed Forces members.

During the second round of cognitive interview, three respondents identified someone who had lived in the household one year ago but were no longer a usual household member at the time of the interview. One respondent reported his parents, another reported his sister-in-law and the third respondent reported a foreign exchange student whom she was hosting. This revised question worked as designed. However, we recommend a minor revision to this item to reflect the changes needed for the UREMIG item.

Final Recommendation for HHEMIG

Other than the people we have listed as household members, did anyone else live or stay here one year ago (about August 12, 2007) but is now living outside the U.S.?

4. WHOOUT(Round 2 only)

Who is it? Anyone else?

To reduce burden for both respondent and interviewer and to improve the instrument efficiency, the names of all household members were listed so the interviewer can check off the name of the person(s) who is/are outside the U.S. and the EMIGLIST question will be skipped. No issue was observed for this question. No change is recommended.

5. EMIGLIST

(Can you tell me who that was?) List the names of ALL people who have left the household to live abroad within the last year.

This question is designed to collect names of person who is outside the U.S. In the first round, we had to type in the names for everyone whether or not the person is a currently household member or has left the household. In the second round, this question is designed to collect names for those who had left the household. This revision has improved the efficiency of this roster procedure. No issue was observed for this question. No change is recommended.

6. EMI_INTRO (Round 2 only)

Now I would like to ask some basic information about [person’s NAME [who is abroad/on EMILIST] specifically.

No issue was observed for this item. However, given the term ‘abroad’ was found to be problematic in later section, the phrase ‘outside the U.S.’ will be used to replace the term ‘abroad.’

Final Recommendation for EMI_INTRO

Now I would like to ask some basic information about [NAME] specifically.

7. EMISEX

Is [EMIG1] male or female?

Nine persons were identified in the two rounds of interview. Four were male and five were female. No issue was observed for this question. No change is recommended.

8. EMIAGE

How old is [EMIG1]? (Your best guess is fine).

All respondents were able to report the ages of the person who was outside the U.S. These people range from 18 to 82 in age. No issue was observed for this question. No change is recommended.

9. EMIREL

How is [EMIG1] related to [Reference Person]?

During the two rounds of cognitive interview, respondents identified a total of nine persons. The reported relationships include mostly relatives (spouse, parent, mother-in-law and other relatives: sister-in-law) and a couple of non-relatives (a roommate and a foreign exchange student.) No issue was observed for this question. However, it is standard practice in almost all of Census survey to provide a followup statement after the relationship question that gives the direction of the relationship to prevent respondent from reporting reverse relationship.

Final Recommendation for EMIREL:

How is [EMIG1] related to [Reference Person]?

[EMIG1] is [Reference Person’s] …

10. EMICOB

In what country was [EMIG1] born?

All but one of the nine persons identified were born outside the U.S. The native born person was studying abroad. No issue was observed for this question. No change is recommended.

11. EMICIT

Is [EMIG1] a citizen of the United States?

Of the eight persons born outside the U.S., only one was a citizen at the time of the interview. Five were visitors of the U.S. and has returned to their home country or traveling in other countries. The other three were traveling for work. No issue was observed for this question. No change is recommended.

12. EMICITB

Did [EMIG1] become a citizen of the United States through naturalization?

The sole person born outside the U.S. had become U.S. citizen through naturalization. No issue was observed for this question. No change is recommended.

13. EMICOR

In what country is [EMIG] living now?

The countries that the nine persons were residing include England, Italy, Madagascar, Peru, Russia and Taiwan, No issue was observed for this question. No change is recommended.

14. EMIAF

Is [EMIG1] a member of the U.S. Armed Forces?

Respondents easily understood this question but interviewers find it somewhat silly to administer this question to seemingly inappropriate target, e.g. an 82 year-old lady. Hence it is recommended that the universe of the question should be limited persons between the age of 18 and 65.

15. EMIGOV

Is [EMIG1] a civilian employee of the U.S. Government?

None of these nine persons was civilian employee. No issue was observed for this question. No change is recommended.

16. EMIAFDEP

Is [EMIG1] a dependent of a member of the U.S. Armed Forces?

None of these nine persons was a dependent of a member of the U.S. Armed Forces. No issue was observed for this question. No change is recommended.

17. EMIGOVDEP

Is [EIMG1] a dependent of a civilian employee of the U.S. Government?

One of these nine persons was a dependent of a civilian employee. No issue was observed for this question. No change is recommended.

18. EMIACT

What is [EMIG1]’s main activity?

Of the nine persons identified, three were working, one was taking care of house or family, two were going to school and the remaining three were retired. No issue was observed for this question. No change is recommended.

2. Transfers Section

As expected, while administering this section, we observed some common privacy concerns similar to those in other surveys that ask about respondents’ personal finance and income. However, despite our initial concern, the majority of our respondents found the questions appropriate and was comfortable with the questions. One respondent thought it was “perfect (for us) to ask” the transfer questions as she reasoned, “it is needed because people don’t realize how important it is for people to be counted for allocation of resources.” However, a few respondents did find the questions, in particular, the amount question, somewhat intrusive. Others with no such concerns had mentioned that the information solicited in this section is more private than those in other sections even though they were comfortable responding to the question. Others commented that their initial incipient concerns were allayed by the initial confidentiality reassurance and explanations of the basic nature of the survey. Below are detailed summary for each of the items tested in this section.

1. TRINTRO (Round 1)

R1 The next part of the interview is about anyone in this household who may have monetary exchanges with family and friends living abroad. By “monetary exchange” we mean the act of giving or receiving money, in US or foreign currency.

Also – we know that people aren’t used to talking about their monetary exchanges, but we ask these questions to get an overall picture of your community and the nation, and NOT to find out about you or anyone in this household personally.

R2 The next part of the interview is about receiving or sending money to family and friends living abroad.

During the first round of cognitive interview, we identified a couple of issues with the introductory statements.

1. Problematic terms: there are two terms that were either overly technical or were unknown to respondents.

   1. “Monetary exchange” was not understood by many of our respondents. Almost half of the respondent during the first round of testing misunderstood the term despite the fact that the definition was provided in the introduction, they were able to define the term correctly. For instance, one respondent said, “it sounds like you are buying stuff.” Another ask the interviewer “Is it how much my monthly expenses is?” Or another thought we were referring to ‘exchanging foreign currency.” One thought of it to mean “any sort of currency” or “anything that is, holds value…land, real-estate, money, currency, jewelry, that has some value, some monetary value”

   2. The term ‘abroad’ was an unknown term to one of the respondent in the second round of interview. Another respondent clue us in that she knew the term when we asked her to define it, but she told us only people in the U.S. use this term. Several of our respondents use the word ‘overseas’ to represent this concept.

2. Overly Complex sentences: The introduction was too long and overly complex for respondent who is less proficient in English. One interviewer reported that her respondent look exasperated after she read the introduction to her. The statements were repeated a couple of times while the respondent was listening to it attentively before she understood the statements. When the interviewer ask the respondent to paraphrase the question, she said “No (I can’t paraphrase). It’s too long. If you want me to understand these questions thoroughly, they have to be very short. The interviewer read the statements a couple of times and the respondent listened to it attentively before she can understand the question. As a result, we recommend reducing the length of the introduction for round 2 testing.

3. Sensitivity. The majority of our respondent did not find the monetary questions inappropriate during our initial round of testing. However, one respondent noted during our debriefing session that the introduction did little to reduce her concern about us asking the questions in the section. In fact, she thought it raises her concern and suspicion of why we are asking these questions.

Recommended revised wording for Round 2 TRINTRO:

As a result of the above findings, we revised the initial introduction by merely introducing what type of questions will be administered in this section and provided optional text for respondents who may needed additional explanatory text of the purpose of the collection of monetary transfers information.

Final Recommendation for TRINTRO

[HOUSEHOLD LEVEL, ASK ONLY ONCE]

The next part of the survey is about money sent to and received from family and friends living outside the U.S., either regularly or for special occasions. We understand this is a common thing.

READ IF NECESSARY.

Also, we know that people aren't used to talking about money. We want to get an overall picture of how families and friends continue to support and keep ties to each other while living in separate countries. We are NOT trying to find out about you or anyone personally.

2. TRSEND

R1 In the last 12 months, has anyone in this household GIVEN or SENT money TO relatives or friends living outside the U.S.?

R2 In the last 12 months, has anyone in this household GIVEN or SENT money TO relatives or friends living outside the U.S.?

READ IF NECESSARY

We know that people aren’t used to talking about their money, but we ask these questions to get an overall picture of how many American households send money to friends and relatives in other countries. We are NOT trying to find out about [you/anyone in this household] personally.

Thirteen of our 25 respondents reported that they have remitted money to family or friends living outside the U.S. We observe one issue for this item.

1. Sensitive question: One respondent refused to answer this question and the interview skipped the entire section for this respondent. As a result, we recommended that the optional text for this question to be made more prominent in the introduction section so that such concerns could be allayed by the optional text earlier.

Final Recommendation for TRSEND

In the last 12 months, did anyone in this household GIVE or SEND money TO relatives or friends outside the U.S.? Please include all money gifts or loans.

[HELP SCREEN]

Friends are people you know personally (are acquainted with). Do not include contributions for charitable organizations.

3. TRSENDNUM

R1 About how many times in total over the last 12 months have members of this household GIVEN or SENT money TO relatives or friends living outside the U.S.? (Your best guess is fine.)

R2 About how many times in total over the last 12 months was that? (Your best guess is fine.)

Twelve out of 13 respondents provided an adequate response to the question. We identified several issues:

1. Recall or Estimation Difficulty: Although most respondent were able to provide an adequate answer to this answer, it took them sometime to figure out an answer given they have to recall everyone in the household who has sent money and when the transfer took place. This is especially difficult for respondents who were providing proxy information; it is difficult for them to provide estimation. However, this is not a major issue given we have made it explicit that we are only requesting a rough estimation

2. Inadequate response options: Given the question allows a respondent to provide estimation, several of them provided a response range. However, this requires interviewer to probe for a single response.

Final Recommendation for TRSENDNUM.

About how many times in total over the last 12 months was money given or sent? (Your best guess is fine.)

4. TRSENDAMT

R1. In the last 12 months, about how much in total dollars have members of this household GIVEN or SENT TO relatives and friends living outside the U.S.? (Your best guess is fine.)

R2. About how much in total dollars over the last 12 months was that? (Your best guess is fine.)

1. Recall or Estimation Difficulty: We observed that recall issue is particularly common when the remittance is not regular. One respondent noted “we don’t keep receipts,” and ‘12 months (reference period) is too long (CS-1).’ Even for respondents who sent money regularly, the respondent found it difficult to remember how much they sent each month for ‘many months ago and do the math.’

2. Sensitive: Although the majority of our respondents (11 out of 13) have willingly and some proudly reported the total amount of money they have sent abroad, a couple of respondents have refused this question. These two respondents have provided valid responses to the two questions leading to the TRSENDAMT item. One respondent provided privacy concerns for refusing to respond to the question; “I’ve come to the conclusion that I am correct in thinking that here is where the government far reaching into the private life of individuals.” We assume a similar reason for the other respondent.

Final Recommendation for TRSENDAMT. Simple recall difficulty is a problem for which we have no recommended solution. As noted above, we have moved the optional text explaining the purposes of the questions to a more prominent place – introduction of the section, we will recommend the following revision mainly for easy of comprehension.

In the last 12 month, about how much in total dollars was given or sent? (Your best guess is fine.)

5. TRREC

In the last 12 months, has anyone in this household RECEIVED money FROM relatives or friends living outside the U.S.?

During both rounds of interview, a total of five respondents reported receiving money from relatives or friends living outside the U.S. Another respondent sent an email to one of our interviewer after the testing and report money she received but has forgotten to report. We have identified three issues for this question.

1. Recall or Estimation Difficulty: A couple of the respondents forgot about money that they have received for special occasion such as birthday and holiday gifts from relatives. For instance, one respondent email one of our interviewer after the completion of the interview that she has forgotten about a large loan she has borrowed from her relatives for her child’s college expenses. Another reported receiving money and gift but later on when asked the TRRECNUM question she recalled only receiving presents over the last 12 months, not money.

2. Ambiguous concept: As described above, although the concept of money is clear to everyone, it is difficult for people to know what type of ‘money’ should be included, e.g. occasional gift or loan seems to be more unclear.

3. Questionnaire effect: Response of question is impacted by previous questions on money sent to family and friends outside the U.S. One respondent didn’t realize we have switched the topic and was still reporting money she has sent home.

Final Recommendation for TRREC

In the last 12 months, did anyone in this household RECEIVE money FROM relatives or friends living outside the U.S.? Please include all money gifts or loans.

6. TRRECNUM

R1 About how many times in total over the last 12 months have members of this household RECEIVED money FROM relatives or friends living outside the U.S.? (Your best guess is fine.)

R2 About how many times in total over the last 12 months was that? (Your best guess is fine.)

Two respondents in the first round reported that they had received monetary gift from relatives living outside the U.S. One respondent’s daughter had received monetary gifts and another respondent had received a large loan form her family. No apparent difficulty was observed with this question in the first round.

In the second round of cognitive pretesting, we were provided with an alternative version of the question, which is shorter than the previous one. Three respondents provided an adequate answer to this question. We did observe one issue.

1. Questionnaire Effects: As noted before, one respondent thought we were still asking about money sent outside the United States because the question wording in the second round omitted the phrase ”…have members of this household received money from relatives or friends living outside the U.S.” The apparent omissions confused the respondent and she

Final Recommendation for TRRECNUM:

About how many times in total over the last 12 months was money received? (Your best guess is fine.)

7. TRRECAMT

R1 In the last 12 months, about how much in total dollars have members of this household RECEIVED FROM relatives and friends living outside the U.S.? (Your best guess is fine.)

R2 About how much in total dollars over the last 12 months was that? (Your best guess is fine.)

During the first round (R1) of the interview, only one respondent was administered this question and she provided an adequate response. All other respondents understood the question very well. No difficulty was observed for this item. However, as noted above, one respondent provided an answer post interview via email to one of our interviewers. So the only issue observed for this item is recall.

In the second round (R2) of cognitive pretest study, we were provided with an alternative version of the question, which is shorter than the previous one. Four respondents provided an answer to this question.

Final Recommendation for TRRECAMT: As noted earlier, our revision to this question focus mainly on communicating the reference period and the context of receiving money, which the latter was missing in the R2 version.

In the last 12 months, about how much in total dollars was received? (Your best guess is fine.)

8. Sensitivity and Appropriateness

During our debriefing session, we inquire respondents whether they think questions in this section were sensitive or inappropriate for a survey. Some respondents provided overall comments, which gives researchers a glimpse to the issue of privacy and confidentiality. Although our initial concerns were that questions in this section were too sensitive and may be consider inappropriate by some respondents, we realize that the concerns expressed by many of the respondents were more general and were not limited to this section per se. One respondent noted that his being uncomfortable with some of the questions has nothing to do with questions administered in this section, but for the general discomfort he had for providing private and personal information when participating in a survey. Another respondent admits that she would have never participated in the interview if it were an actual survey. She would not have provided much of the information at all. The only reason she agrees to the interview was because her sister works at the Census Bureau and she knows about the interviewer by reference where she felt safe to disclose private information. “Immigrants like us, even if we have been here for a while, we still don’t want others to know our situation. I want to protect myself (private information)...” A comment from another respondent on this issue was somewhat encouraging. “I can always refused to answer some questions.’ In sum, we believe the questions designed for this section was appropriate and covers the same ground as the main CPS survey and we anticipate item nonresponse to be higher than generic demographics questions, similar to other survey items that asks about personal finance. Any important dismay about survey content is much more likely to arise from the main survey itself, as opposed to the monetary transfers questions. Hence, we doubt that administering these questions will risk increased attrition to CPS. However, we do recommend that future large-scale field implementations of the Migration supplement series remain alert to the possibility of negative impacts on cooperation.

5. Summary and Conclusions

In this section, we will summarize the cognitive results based on each of the five goals set for this research and draw broad conclusions from the results.

1. First, one of the major goals for this study is to examine respondents’ comprehension of the newly proposed question; whether they can understand the question wording and its intents. Below are several major findings.

   1. Concepts of live and stay: these terms are ambiguous; most people can distinguish the two concepts providing their own definitions of what each meant. Although most respondents have a clear distinction between the term staying and living, where ‘to stay’ is temporary such as ‘visiting or ‘vacationing’; the term ‘to live’ is more permanent concept and is tied to the notion of permanent residence, home ownership, adaptation, and involvement in the community. However, for the purpose of our data collection, the two terms have to be used carefully and sparingly. Two respondents said it themselves; the two terms can be the same.

      1. “Feels like it’s almost the same thing, interchangeable. You can live here and at the same time you are staying here, and vice versa.”

      2. “The point is whether I’ve ever entered the country before. It doesn’t matter how long. One time it is 2 months, the other time is in general. The meaning to me is the same.” This respondent was referring to the two questions INUSYR and YOEFIRSTCHK.

2. Questions’ length. It is important to keep in mind when a survey is designed to study respondents whose English is not their first language, and when a translated survey is not possible, the length of questions should be kept short. During our debriefing sessions, several respondents have provided such sentiment. “In a survey, questions should not be too long." Another provided a similar reason for shorter question. “Some questions are very tricky, long sentences make me..sometimes I am confused.” We encounter once incidence where the interviewer had to terminate the interview shortly after the respondent completed the demographic sections because the respondent needed language assistance and the interviewer had to translate the questionnaire ‘on-the fly.’ We did not include this interview in this report other than noting the fact that CPS respondents, who are not fluent in English, may not understand the current question series tested in this study. We recommend that this new supplement be translated and pretested, at a minimum, in Spanish, by following the Census Bureau’s best practice for translated survey.

3. Vocabulary. Similarly, vocabularies used for such a survey should be as simple as possible so that most ESL respondents can understand our question intent and meaning.

4. Idea of rough estimates: we use the phrase; “your best guess is fine” at the end of numerous questions to communicate that we understand we accept uncertainty in some of our answers and are seeking the best estimated answers respondents can provide. (TIMABTOT, TIMABDYR, OUTYRLASTYR, YOELAST; EMIAGE, TRSENDNUM, TRSENDAMT, TRRECNUM & TRRECAMT)

2. Our second goal was achieved. We found that the four sections flow well together and it is operationally feasible to implement all the proposed questions in the supplement in the order they were presented. In particular, we found the placement of the transfer section to be ideal given the section does raise some privacy concern for a few respondents. Hence, it is recommended that the Transfer section be kept at the end of the supplement so that it will not inadvertently affect the item response rate for other sections.

3. Thirdly, we did observe some difficulties for respondents (most likely proxy respondents) to recall relevant information to answer the questions; however, none were major.

4. Lastly, we found that most respondents find the questions on monetary transfer appropriate; and the introductory statement explaining the purpose of asking the monetary transfer questions helps to reduce privacy concern, if any.

6. Future Research

We offer the following two items for future research.

1. Pretest revised wording for the INUSYR question “When is the first time [You/NAME] come to the U.S. for 2 months or more?” This recommended version avoids the usage of problematic and ambiguous terms such as ‘live’ or ‘stay’ by stating the length of time researcher consider as ‘live’ or ‘stay’.

2. Provide a translated version of the supplement and conduct cognitive pretesting on the translated survey.

7. Acknowledgement

The author would like to thank her colleagues from Population Division (POP), Demographic Statistics Division (DSD) and Statistical Research Division (SRD) for their assistance with this research. This includes Melissa Chiu (POP), the project manager and sponsor for this project, who provided valuable comments throughout the entire research project; Patricia de la Cruz and Cassandra Logan (POP), and Kristin Hanaoka, Vicky Smith, Chris Stringer and Kenya Tyndle (DSD) who conducted three quarter of the interviews and prepared detailed summary reports; Loraine Randall (SRD) who carried out the challenging tasks of recruiting cognitive interview laboratory subjects. I am grateful to their contributions for the success of this project.

8. References

Wellens, Tracy. “The Cognitive Evaluation of the Nativity Questions for the Current Population Survey.”

1