SSB - Mode Testing of Online Annual Survey of Refugees Mode

Mode Test of Online Annual Survey of Refugees

Generic for Pre-testing of ACF Data Collection Activities

0970 - 0355

Supporting Statement

Part B

October 2024

Submitted By:

Office of Refugee Resettlement

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Overview of Study Objectives

The Office of Refugee Resettlement (ORR) funds an Annual Survey of Refugees (ASR) (0970-0033) to collect information on refugees’ progress toward self-sufficiency and integration during their first five years of living in the United States. The survey data are used to meet reporting requirements for ORR’s Annual Report to Congress and inform refugee resettlement policy, by providing a unique source of nationally representative information on refugees’ experiences of adjustment to life in the United States. The ASR collects a range of information about refugees, including language proficiency, training and education, employment and income, public assistance, social connections, and access to health care. The annual survey is conducted by telephone in up to twenty languages.

ORR is interested in transitioning to a multi-mode version of the survey, in which respondents could participate either by telephone or online. This proposed data collection will inform those efforts. This study is a mode test of an online option to determine whether a multi-mode approach is possible in future versions of the ASR.

The overall design of this project will be conducted by the Urban Institute (UI), along with one subcontractor, SSRS. SSRS will be responsible for programming the online survey, selecting the study sample from the administrative data, updating contact information, recruiting participants, and providing data on survey participation.

B1. Respondent Universe and Sampling Methods

Target Population

The target population is refugees resettled to the U.S. in the past five years, which is a diverse group coming from a variety of national, ethnic, and educational backgrounds.

For purposes of feasibility for the mode test, we will limit the study population to Principal Applicant refugees who arrived between FY2019 and FY2023, excluding those who were already sampled for the telephone-based ASR, those who are under eighteen years old, and those who do not speak one of the six languages selected for the mode test (see Table 1), i.e. Arabic, Kiswahili, Ukrainian, Dari, Spanish, and Sgaw Karen.

The first four languages were used in the phase one feasibility test conducted in 2023, and the two additional languages were selected to capture geographic variation and include languages that may be numerous in future refugee admissions. According to estimates of the top 20 languages for the 2024 ASR in table 1, all six languages are among the top 20 languages spoken by recently resettled refugees. Bolded languages were chosen for the mode test.

Sampling Methods

Study participants will be drawn from administrative data, specifically an ORR database that captures all refugee arrivals, the Refugee Arrivals Data Systems (RADS).

This mode test will utilize a stratified random sampling approach, selecting the sample from the same sampling frame as the 2024 ASR (refugees who arrived between FY2019 and FY2023). The sampling approach will mimic the telephone-only-ASR sampling approach, which uses a number of factors for stratification: cohort (cohort 1: 2019-2020; cohort 2: 2021-2022; and cohort 3: 2023 arrivals), year of arrival (for cohorts 1 and 2 only); geographic region of origin (Africa, East Asia, Europe, Latin America/Caribbean, Near East/South Asia), native language (Arabic, Kiswahili, Ukrainian, Dari, Spanish, Sgaw Karen), age group (18-29, 30-44, 45-64, 65+), gender, and household size (1, 2, 3+ persons). We will utilize proportionate stratified sampling within cohorts toensure that the resultant sample is highly representative of the refugee population.

With a goal of reaching 100 survey respondents, we plan to start with a sample of 2,000 people. We will use a replicated sample design, which is a large random sample that is randomly partitioned (using the same stratification scheme as the original sample) into many smaller samples, each of which is a snapshot of the original large sample. Releasing sets of replicates rather than the entire sample at once allows us to only work as much sample as is needed to complete the target of 100 online surveys.

Table 1: Estimated Coverage of 2024 ASR Refugees – Primary Language Spoken

Language Count	Primary Spoken Language	Primary Spoken Language %	Primary Spoken Language Cumulative %
1	Arabic	17.6 %	17.6 %
2	Kiswahili	17.0 %	34.6 %
3	Kinyarwanda	8.0 %	42.5 %
4	Spanish	7.6 %	50.1 %
5	Dari	6.4 %	56.5 %
6	Ukrainian	4.8 %	61.3 %
7	Kibembe	3.7 %	65.0 %
8	Russian	3.5 %	68.5 %
9	Rohingya	3.0 %	71.5 %
10	Sgaw Karen	2.8 %	74.3 %
11	Burmese	2.1 %	76.4 %
12	Kinyamulenge	2.0 %	78.5 %
13	Tigrinya	2.0 %	80.5 %
14	Somali	2.0 %	82.5 %
15	Farsi	1.6 %	84.1 %
16	Kinyabwisha	1.4 %	85.5 %
17	Tedim	1.0 %	86.5 %
18	Farsi, Western	0.5 %	87.1 %
19	English	0.5 %	87.6 %
20	Kayah	0.4 %	88.0 %
21-205	Remaining 185 languages	Not Covered for 2024 ASR

*Bolded languages were chosen for the mode test

Appropriateness of Study Design and Methods for Planned Uses

The information gathered in this mode test is not intended to be used as a basis for public policy decisions and is not expected to reach the threshold of influential or highly influential scientific information. Rather, it is intended as a test of whether an online mode for the ASR is likely to yield valuable information that may not be able to be obtained in the traditional telephone mode or can be collected at a lower cost or with a lower burden for some participants.

B2. Procedures for Collection of Information

Data Collection Processes

SSRS will use addresses and telephone numbers from the administrative data that have been updated through standard tracing databases proven effective for prior ASRs. SSRS will recruit potential respondents through a four-tiered mailing design: with an invitation letter, a sealed reminder postcard, a reminder letter, and a final postcard, all of which will be translated into the six mode test languages. Like the ASR, the invitation letter will contain a $2 preemptive token of appreciation. All mailings will contain a link and QR code to the web survey, a unique passcode for the respondent to use to access the web instrument, and information on how respondents can contact support if they have any questions. The online survey will begin with several screening questions to ensure the sampled individual is indeed taking the survey.

All refugees who complete the online survey will receive a $30 gift card, like respondents to the telephone-based ASR.

Data Handling

The project team will use a secure data file transfer process using Secure File Transfer Protocol (SFTP). The data received from HHS/ORR (RADS administrative data) is transmitted to Urban through an HHS SFTP. Subsequent transfers, including the RADS and ASR survey data, between UI, HHS, and SSRS are transmitted through the UI Secure Data Site, which is an on-premise SFTP server behind the UI firewall. The UI SFTP will be used to transmit sample data between SSRS and RSS, but UI will not download these data. UI and SSRS will abide by the Data Security and Monitoring Plan that was approved for the ASR to protect the personally identifiable information of sampled refugees and respondents.

Survey responses will not be analyzed other than for purposes of assessing data quality.

Data Use

UI will determine whether the data from the test indicate whether an online version of the ASR should be implemented in the future from both a technical and value perspective and will share those findings with ORR.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Response Rates

This is the first large-scale test of an online-only version of the ASR, and it is not known what the response rate will be. For the telephone-only ASR. we normally achieve a response rate around 20 percent , and this varies across languages and other demographic characteristics such as age and cohort (amount of time in the U.S.). We expect that the response rate will be lower for an online-only version. Based on prior research in which we asked refugee respondents about their mode preference, we expect that an online version may be appealing to certain refugee subpopulations with high educational attainment and digital literacy, but that most resettled refugees who have limited education and literacy would prefer a telephone version. This mode test will include a variety of demographic profiles, and we will be examining response rates and item nonresponse by language and other factors.

Maximizing Response Rates and Dealing with Nonresponse

The four-stage mailing recruitment process is designed to maximize response rates. This process, including a translated invitation letter, sealed reminder postcard, reminder letter, and final postcard, will encourage potential participants to consider participating on multiple occasions, and they will be reading concise and slightly modified language each time. This is all designed to maximize the likelihood of someone choosing to participate.

Access to taking the online survey will be facilitated through inclusion of a QR code as well as a weblink and unique passcode in all mailed materials.

In addition, a telephone line will be provided to support individuals who have questions about the survey and want to speak with a person in their native language. The telephone support number will appear not only on outreach materials but also on the survey screen. Our expectation is that this assistance option will help participants successfully navigate and complete the survey.

Like with the ASR, the invitation letter will contain a $2 preemptive token of appreciation, and the informed consent and engagement materials also promise a $30 post-participation token of appreciation, which will be sent to participants who complete surveys. There is strong research evidence that these measures increase response rate.¹

B4. Tests of Procedures or Methods to be Undertaken

The purpose of this information collection request is to test a different method for data collection instruments and survey procedures. HHS expects that all data gathered under this clearance will result in improved survey procedures and thus reduce respondent burden and increase data quality.

B5. Individual Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

HHS has contracted with UI to design the mode test, organize the data collection, and analyze the data. The Principal Investigator of the study at UI is Hamutal Bernstein, Senior Fellow, who can be reached at [email protected] or 202-261-5840.

Attachments

• Letters of invitation and reminders

• ASR online instrument

1 See Rao, Neomi. 2020. “Cost Effectiveness of Pre- and Post-Paid Incentives for Mail Survey Response.” Survey Practice 13 (1). https://doi.org/10.29115/SP-2020-0004.

7