Part B PIAAC Cycle 2 2022 Operational Field Test

The primary goals of the PIAAC Cycle II Operational Field Test are to collect high-quality data that support the assembly of the questionnaire and direct assessment instruments, the functioning of the tablet-based platform, and the survey operations in preparation for the Cycle II Main Study in 2022. The Operational Field Test will utilize volunteers in metropolitan Washington DC and will provide a thorough test of questionnaire and assessment implementation. It will test the functionalities of the tablet device and ease of use of the device and interface for adults with low levels of familiarity with tablet devices. The test will provide information about interviewer difficulties related to survey administration and also improvements that may be needed for interviewer training for the Main Study. A convenience sample of volunteers in the greater Washington DC metropolitan area will be recruited to participate. Volunteers will be administered the PIAAC interview components in their entirety, including a screener questionnaire, the Background Questionnaire and an assessment of numeracy or literacy skills.

In addition to the Operational Field Test meeting operational goals, respondents’ data will contribute to international analyses that will examine the fit of the international parameters for cognitive items (numeracy and literacy skills) across the different participating countries (and different country/language versions when possible). These analyses allow the estimation of the level of comparability among the items and scales. Items that do not show good and consistent fit across the majority of country/language versions are candidates to be dropped from the Main Study instruments. The data from various countries will allow for a more stable estimate of the international comparability of all cognitive items and best supports the decisions made in assembling the Main Study instruments.

The overall sample yield goal for the Operational Field Test is to reach 250 16-65 year olds. The Main Study target population will also include 66-74 year olds, but due to particular risk to this population from the novel coronavirus (COVID-19) pandemic, this group will not be part of the Operational Field Test. The 66-74 year olds are of special interest to the U.S. but are not part of the international comparison group. Selection and assessment of 66-74 year olds was tested in Cycle I of PIAAC.

Three prior U.S. PIAAC data collections have occurred to date in Cycle I. The United States completed its PIAAC Cycle I 2012 Main Study along with the 23 countries who participated internationally. It included a sample of 5,010 adults in 80 Primary Sampling Units (PSUs). The survey components included a screener, an in-person background questionnaire, and a computer-based or paper assessment. A second U.S. data collection, the PIAAC Cycle I 2014 National Supplement of 3,660 respondents, was conducted with the same survey components. The PIAAC Cycle I 2012 and 2014 data collection efforts together formed an enhanced 2012/2014 combined nationally representative sample. The PIAAC Cycle I 2017 National Supplement was the third data collection for PIAAC in the U.S. National estimates of proficiency in literacy, numeracy and problem solving from the 2017 national sample provided additional set of data points for evaluating trends in proficiency as compared to estimates from the 2012/2014 combined sample, and earlier adult literacy surveys. In addition, the data collected from all survey years (2012/2014/2017) were used to form sub-national model-based estimates through small area estimation (SAE) methods.

B.1 Volunteer Sample Design

The target population for the Operational Field Test consists of 250 adults age 16 through 65 years. To achieve a diverse sample, volunteers will be screened on age, gender, education, and race/ethnicity. Ensuring variation in education, English language proficiency and computer/tablet skills will help test functionality across volunteers with varying skill and education levels. The questions used in screening volunteers will be based on the PIAAC Cycle II Screener and Background Questionnaires. Table B.1 displays the target quotas by category for recruitment, which are based on 2019 Current Population Survey totals in these categories.

Table B.1 PIAAC Cycle II Operational Field Test Recruitment Quotas

Item	Total
Gender
Male	125
Female	125
Age
16-25	50
26-35*	50
36-45*	50
46-55*	50
56-65	50
Education
Some High school	35
HS Degree	65
Some College*	70
College Degree or higher*	80
Ethnicity
Hispanic	50
Not Hispanic	200
Race
White	195
Black/African American	35
Asian*	17
Native Hawaii/Pacific Islander*	1
American Indian Alaskan Native*	2
English Language (First Language)
English	215
Not English	35
[Digital device use] “In everyday life, how often do you usually us a digital devices, such as a smart phone or tablet outside work?”
Never	15
Less than 1 time month	45
Less than 1 time per week, at least 1 time per month*	45
At least 1 time per week, not every day*	45
Everyday	100

*Rows may be collapsed within characteristic as necessary to meet category quotas by data delivery deadline.

Within categories, quotas will be set by total, with a tolerance level of +/- 20% for each category, keeping the overall number at 250. These targets accommodate the need to efficiently capture diverse groups of respondents within a five-to-six-week field period, and some quota targets within categories may be combined as necessary (indicated by the asterisk).

B.2 Procedures for the Collection of Information

Contacting, Screening and Interviewing on Cycle II U.S. PIAAC

Recruitment announcements will advertise the research opportunity and timeline in a variety of places including the Westat corporate website, online bulletin boards such as Craig’s List and other social media. If necessary, to achieve a diverse set of respondents who meet the criteria outlined in the section B.1, local population-based organizations may be engaged (e.g. Hispanic Chamber of Commerce, National Black Chamber of Commerce) to advertise the study among their constituents. These announcements will direct volunteers to a toll-free number where they will be screened for eligibility (the screening script and other recruitment materials are shown in Appendix G). Once quotas are met within each category, volunteers with those characteristics will no longer be eligible to participate. Interviews will be conducted in a location with established COVID-19 mitigation procedures, including temperature scans, use of masks/face shields, social distancing and sanitizing of all materials and equipment. Respondents will be informed that they will receive a $100 incentive upon completion of the questionnaire and assessment. The incentive amount considers the time and expense of travel to the interview facility. A sample recruitment announcement is included below.

Sample Recruitment Announcement

Westat is seeking volunteers to participate in the Program for the International Assessment of Adult Competencies study. To be competitive in the world economy, the U.S. must have a knowledgeable and skilled workforce. This study provides important information about the education, training and skills Americans use on the job and in everyday life.

We need people like you to participate in this important effort.

You will be asked to complete an interview and exercise with one of our field representatives.

The interview involves some reading and you will get to do some interactive problem solving.
We will also ask about different skills you have used when employed.

The interview and exercise will take about 90 minutes to complete, all information will be kept confidential, and you will receive $100 for participating in the PIAAC field test. If you would like to participate, please call 1-888-xxx-xxxx to see if you are eligible to volunteer for this study.

Interview Administration

Pre-Interview Protocol

Interviews will be conducted in a location with established COVID-19 mitigation procedures, as adapted from the Westat Interviewer Procedures Manual (Appendix E). These procedures include the following pre-interview assessments and briefings:

Confirmation Calls the Day Before the Interview Interviewers will call to confirm appointment for the following day and ask a series of COVID screening questions, found (Found in Section 2.4 of Appendix E). If volunteers respond “yes” to any of these questions above, schedule a time to call back after the 14 day period.
Interviewer Self-Assessment Interviewers and research staff will be required to monitor their health while working on their assignments, which includes completing COVID-19 screening prior to the start of the workday. The Checklist can be found in Appendix E, Section 1.1.
On-site Safety Protocol Review. Interviewers will greet volunteers at the entrance to the research facility and review the safety protocol for the interview session. This includes the use of masks, conduct of the interview in outdoor space or socially distanced area, gloves, sanitizing procedures for the sharing of the tablet and other study materials.

Interview Protocol

Introduction and Questions: Interviewers will begin each session by reviewing the purpose of the field test, following a script that explains the purpose of the field test and what will happen during the session. In this introduction, interviewers explain that they will walk through each component of the study to mimic administration of the screener, background questionnaire and exercise just as they would do so in the PIAAC Cycle II Study in households. The volunteers will be given the opportunity to ask questions. See Appendix A1for this introduction.
Field Test: Interviewers will then proceed to conduct the interviews as they would do so for the Main Study in sampled respondents’ households. As in the Main Study, the session begins by providing a copy to the volunteer respondent of the Advance Letter to introduce the study, as is done for the Main Study.

Screening Instrument. Although the volunteer will have already answered screening questions during recruitment, the interviewer will administer the existing PIAAC Cycle II screening instrument and reiterate that it is being administered for testing purposes.
Background Questionnaire. The interviewer will then proceed through the Background Questionnaire which is Interviewer Administered and takes approximately 45 minutes to complete. Questions cover education, job history, job skills training as well as some general opinion questions. Interviewers will provide a booklet of ‘showcards’ showing the response categories for each question for the volunteer to reference¹. This is a disposable paper booklet handed once to the respondent. The background questionnaire is included in Appendix C.
Exercise. Upon completion of the background questionnaire, the interviewer will sanitize the tablet and hand it once to the respondent. The Exercise has 5 distinct sections; Tablet Skills, Components, Tablet Tutorial, Locator and Assessment. During the Tablet Skills section, respondents can watch short animations of how to tap, drag and drop, and highlight. They can practice doing each of these actions independently until they are comfortable before they move to the next section of the Exercise.
Incentive. Upon completion, volunteer respondents will be provided with the monetary incentive in the form of a gift card.

B.3 Maximizing Response Rates

This Operational Field Test uses volunteers therefore response rates are not applicable. The goal is to meet the target yield of 250 volunteer respondents. Recruitment will be ongoing until this goal is met.

B.4 Tests of Methods and Procedures

This data collection is being used to test operations and survey implementations. No formal experiments or tests of methods or procedures are planned at this time.

B.5 Individuals Consulted on Study Design

The following are responsible for the statistical design of PIAAC:

Holly Xie, National Center for Education Statistics; and
Stephen Provasnik, National Center for Education Statistics.

The following are responsible for sampling activities:

Leyla Mohadjer, PIAAC Consortium/Westat; and
Tom Krenzke, PIAAC Consortium/Westat.

Analysis of assessment items will be performed by Usama S. Ali and Frederic Robin, Educational Testing Service.

1 A reference to the ‘showcards’ can be seen in Appendix C, page 55 of the Background Questionnaire.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	PIAAC OMB Clearance Part A 12-15-09
Author	Michelle Amsbary
File Modified	0000-00-00
File Created	2021-03-18