Part A PISA 2012 Validation Study 2015

The Program for International Student Assessments (PISA) is an international assessment of 15-year-old students sponsored by the Organization for Economic Cooperation and Development (OECD). In the United States, it is conducted by the National Center for Education Statistics (NCES), of the Institute of Education Sciences, U.S. Department of Education. PISA was first administered in 2000 and is conducted every three years. As part of PISA’s most recent administration, in 2012, PISA’s fifth cycle, a nationally-representative sample of approximately 5,200 15-year-old students took the PISA assessments in the United States (OMB# 1850-0755).

PISA provides information about student performance in mathematics, science, and reading literacy, as well as problem-solving in the United States and more than 65 other participating countries and education systems. PISA is designed to assess how well students nearing the end of compulsory schooling are prepared for the challenges of further education and the workforce. To do so, PISA measures students’ literacy (knowledge and skills to solve problems in real-life contexts) and collects information about students’ home background, attitudes, and experiences in and out of school through a questionnaire completed by students. Furthermore, principals of school attended by the participating students complete a questionnaire about school policies and practices, providing additional context for the student measures.

NCES proposes to conduct a follow-up study with students who participated in PISA 2012¹ to learn how performance on PISA relates to subsequent outcomes and skills of young adults. The follow-up study—referred to in materials to potential respondents as the PISA Young Adult Follow-Up Study, and in this request in short as the PISA Validation Study — will provide information about how students’ skills and experiences at age 15, collected through PISA, relate to subsequent literacy, numeracy, and problem-solving skills, as well as educational attainment, education and work experiences, skills used in daily life, career intentions, and aspects of health and well-being. In fall 2015, when these students will be 18 years of age, they will be asked to take the web-based version of the OECD’s Program for the International Assessment of Adult Competencies (PIAAC) assessment and background questionnaire—the Education and Skills Online Assessment (ESO).

In November 2013, OMB approved the location and tracking of the PISA 2012 respondents (OMB 1850-0900 v.1) to prepare for this follow-up study. This request is to (1) recruit the PISA 2012 sample respondents who have been successfully located and tracked; (2) administer ESO to a field test sample in the summer of 2015; and (3) administer ESO to a main study sample in the fall of 2015. This submission describes the study rationale and design and presents the sample members recruitment materials (Appendix A) and the ESO background questionnaire (Appendix B).

A. JUSTIFICATION

A.1 Importance of Information

As part of a continuing cycle of international education studies, the United States, through the National Center for Education Statistics (NCES), is participating in several international assessments and surveys. The Program for International Student Assessment (PISA), sponsored by the Organization for Economic Cooperation and Development (OECD), is one of these studies (OMB# 1850-0755).

In light of growing concerns related to international economic competitiveness, the changing face of our workplace, and the expanding international marketplace in which we trade, knowing how our students and adults compare with their peers around the world has become an even more prominent issue than ever before. Nationwide, interest in understanding what other nations are doing to further the education achievement of their populations has increased beyond simple comparisons. Data at critical points during the education career of our students help inform policymakers in their efforts to guide and restructure the American education system. These critical points may occur during primary, secondary, or tertiary education, as well as extend into adult education and training programs. Consequently, generating comparative data about students in school, at the end of formal schooling, in postsecondary education, and about adults in the workplace and in the community has become an important focus for NCES.

PISA is part of the larger international program that NCES has actively participated in through collaboration with, and representation at, the OECD, the Asia-Pacific Economic Cooperation (APEC), and the International Association for the Evaluation of Educational Achievement (IEA). Collaboration with Statistics Canada, Eurostat, and ministries of education throughout the world helps to round out the portfolio of data NCES compiles. The United States has participated in PISA since its first administration in 2000.

PISA measures students' knowledge, skills, and competencies primarily in three subject areas –mathematics, science, and reading literacy. The overall strategy is to collect in-depth information on students’ capabilities in one of these three domains every three years so that detailed information on each becomes available every nine years. During each three-year survey cycle, the major focus is on one content domain, with a minor focus on the other two content domains. The major focus for the PISA data collection in 2012 was on mathematics literacy, with a minor focus on science and reading. PISA 2012 also included an assessment of problem-solving and financial literacy.

Investigating the link between PISA and young adults’ preparedness for life and work

PISA is designed to measure “how well students, at age 15, are prepared to meet the challenges they may encounter in future life” (OECD 2009, pg. 9). PISA does this by identifying, with input from participating countries and subject-matter experts, the key competencies that are believed to contribute to students’ future success, and by measuring students’ knowledge and skills in these areas. In short, PISA is designed to assess how well students nearing the end of compulsory schooling are prepared for further education or entry into the workforce and, more fundamentally, to contribute to society as functioning young adults.

Unlike other large-scale international assessments that are based on school curricula, such as the Trends in International Mathematics and Science Study (TIMSS) or the Progress in International Reading Literacy Study (PIRLS), PISA assessments are designed to reflect what students have learned both in and out of school and to measure students’ ability to apply that knowledge to real-world situations. For this reason, PISA has a unique validity challenge. TIMSS and PIRLS assessments can be validated by evaluating how well they cover mathematics, science, and reading curricula in the participating countries. However, comparing PISA measures with content standards or school curricula does not provide evidence that PISA assesses the competencies that matter for students’ “future success” and the “challenges they may encounter in the future.” To validate PISA measures, it is necessary to examine whether the competencies assessed by PISA predict future success.

Given the growing prominence of PISA in the United States, it is of policy interest to better understand how well PISA measures students’ preparedness for further education and the workforce. PISA is widely cited as evidence that U.S. students are not as prepared for the global economy as their peers in other high-performing countries, and such claims rest on the assumption that PISA is measuring the skills that students will need to succeed in the 21^st century and the global economy.

The PISA Validation Study

NCES is requesting approval to conduct a follow-up study with students who participated in PISA 2012 to learn how performance on PISA relates to subsequent outcomes and skills of young adults. The follow-up study—referred to in materials to potential respondents as the PISA Young Adult Follow-Up Study (YAFS), and in this clearance request in short as the PISA Validation Study—will provide information about how students’ skills and experiences at age 15, collected through PISA, relate to subsequent literacy, numeracy, and problem-solving skills, as well as educational attainment, education and work experiences, and skills used in daily life. This will be accomplished by administering a web-based version of the OECD’s Program for the International Assessment of Adult Competencies (PIAAC). PIAAC will be used to measure the skills, outcomes, and experiences of students who previously participated in PISA. The PIAAC assessment, administered in 25 countries, focuses on cognitive and workplace skills deemed necessary for successful participation in 21^st-century society and the global economy (OECD 2012). Specifically, PIAAC measures relationships between individuals' educational backgrounds, workplace experiences and skills, occupational attainment, use of information and communications technology, and cognitive skills in the areas of literacy, numeracy, and problem solving.

The sample for the PISA Validation Study will consist of the approximately 4,700 15-year-old students from the PISA 2012 national sample who took the PISA mathematics, reading, and science test and completed a Student Information Form with their contact information. PISA 2012 recruitment materials, including materials for parents, stated that students supplying contact information may be contacted by NCES for a future study. In fall 2015, when these students will be approximately 18 years of age, they will be asked to take a web-based version of the PIAAC assessment and background questionnaire developed by the OECD—the Education and Skills Online Assessment (ESO). The ESO is described in more detail below. A field trial of the Validation Study is currently planned for September 2015 with a sample of 200 PISA 2012 students who provided contact information but took financial literacy test booklets and are not the target for the PISA Validation Study main data collection.

PIAAC was administered to a national sample of adults 16 to 65 years old in 2011-12. It examined adults’ literacy and skills in the information age, focusing on what are deemed key skills for individuals to participate successfully in the economy and society of the 21st century (Goodman, Finnegan, Mohadjer, Krenzke, and Hogan 2013). It assessed adults’ basic reading skills, reading literacy, numeracy, and problem solving in technology-rich environments (on or with a computer), and measured their ability to use computer and web applications to find, gather, and use information, and to communicate with others. PIAAC used a “Job Requirements Approach” to ask employed adults about the types and levels of a number of specific skills used in the workplace, including not only the use of reading and numeracy skills on the job, but also the use of physical skills (e.g., stamina and manual dexterity), people skills (e.g., public speaking, negotiating, working in a team), and information technology skills (e.g., using spreadsheets, writing computer code). PIAAC also asked about adults’ backgrounds, work and educational experiences, including the requirements of their main job in terms of the intensity and frequency of the use of the aforementioned skills.

The web-based version of the PIAAC assessment and background questionnaire (ESO) will be distributed by the OECD for use by countries, institutions, and individuals to support research and workforce development and training efforts. The ESO consists of two main components that are administered together: an assessment and a questionnaire. The ESO assessment, which is currently being finalized for distribution to countries by the OECD, was piloted in multiple countries and has been equated to PIAAC, allowing for international benchmarking. The ESO questionnaire is based on that used in PIAAC.

Administering the ESO to PISA 2012 participants

In the PISA Validation Study, the ESO assessment and background questionnaire will be administered to sample respondents in 2015 when they are approximately 18 years old, three years after they took PISA in 2012. The assessment is designed to take on average 90 minutes to complete and the background questionnaire approximately 35 minutes, for a total of 125 minutes on average per respondent.

The ESO assessment measures a set of cognitive and non-cognitive skills deemed necessary for full participation in modern societies. These skills and knowledge include being able to understand and use printed and electronic texts, reason with numbers, and solve problems using information and communication technologies (see exhibit A-1, below, for an overview of the cognitive content of the ESO).

Exhibit A-1. Summary of the cognitive domains in the OECD Education and Skills Online (ESO) Assessment

	Literacy	Numeracy	Problem solving in technology-rich environments
Definition	Ability to understand, evaluate, use and engage with written texts to participate in society, achieve one’s goals, and develop one’s knowledge and potential.	Ability to access, use, interpret and communicate mathematical information and ideas in order to engage in and manage the mathematical demands of a range of situations in adult life.	Ability to use digital technology, communication tools and networks to acquire and evaluate information, communicate with others, and perform practical tasks.
Content	Texts are characterized by Medium: Print-based Digital Format: Continuous or prose texts (narration, argumentation, or descriptions) Non-continuous or document texts (tables, lists, graphs) Mixed texts (combination of prose and document elements) Multiple texts (juxtaposition or linking of independently generated elements)	Mathematical content, information and ideas: Quantity and number Dimension and shape Pattern, relationships, change Data and chance Representations of mathematical content: Objects and pictures Numbers and symbols Diagrams, maps, graphs, tables Texts Technology-based displays	Technology: Hardware devices Software applications Commands and functions Representations (text, graphics, video) Nature of problems: Intrinsic complexity (number of steps, alternatives required for solution, complexity of computation and/or transformation, number of constrains) Explicitness of the problem statement (largely unspecified or described in detail
Cognitive Strategies	Access and identify Integrate and interpret (relating parts of text to one another) Evaluate and reflect on	Identify, locate or access Act upon and use (order, count, estimate, compute, measure, model) Interpret, evaluate and analyze Communicate	Setting goals and monitoring progress Planning Acquiring and evaluating information Using information
Context	Personal Work-related Community Education	Everyday life Work-related Society & Community Education	Personal Work-related Community

Source: http://www.oecd.org/site/piaac/ENGLISH_Brochure%20Education%20and%20Skills%20Online.pdf.

The ESO questionnaire, in turn, asks about personal and home background and education and workforce experiences (all taken from the PIAAC questionnaire), and includes modules that were developed and piloted by the OECD for ESO. The PIAAC background questionnaires were more extensive than what will be included in the ESO, however. As shown in exhibit A-2, below, the content of the ESO questionnaire modules focus on four areas: (i) basic demographics, (ii) career interests and intentionality (CII), (iii) behavioral performance competencies (BPC), and (iv) subjective well-being and health (SWBH). The OECD has designed the ESO such that all items and modules must be administered intact; there is no mechanism for the removal or suppression of individual items or modules.

Because the ESO must be administered as designed, NCES is planning to administer an ‘add-on’ questionnaire module, with 16 additional items administered in PIAAC but not included by the OECD in ESO. These additional questions address students’ recent and current educational status and experiences, including non-formal learning activities. This information is critical for realizing the aims of the PISA Validation Study, which is to examine the relationship between PISA and the paths that young adults choose after leaving high school. The themes covered by these additional survey items are also shown in exhibit A-2. More detailed information on all of the non-cognitive survey items is included in Appendix B. Please note that because the ESO modules are designed to be administered online, detailed information on the survey items provided in Appendix B are shown in tables, as there is no paper version of the questionnaire.

Exhibit A-2. Summary of the non-cognitive modules in the OECD’s Education and Skills Online (ESO) Assessment

Non-cognitive survey module	Content themes
Background + Educational Experiences (Core)	Age Sex Highest level of education attained Country of origin	Language spoken at home Native language Current employment status Current occupation
Career Interest and Intent (CII)	Inventory of vocational interests Intent to seek employment	Intent to seek job training Job seeking behavior
Behavioral performance competencies (BPC)	Well being Self-control Responsibility Sociability Curiosity Cooperation Generosity	Adjustment Intellectual efficiency Ingenuity Achievement/Goal setting Dominance Organization and Order
Subjective well-being and health (SWBH)	Life satisfaction Sleep patterns Physical activity	Emotional states Nutrition (diet) Body measures (height, weight)
Additional items not already included in ESO	Current educational trajectory Degree/certification being sought General category of degree/certification being sought Non-formal learning activities (distance education; seminars)	Additional learning activities Reasons for non-participation in learning activities Use of tutors and participation in apprenticeship programs

Rationale for including CII, BPC, and SWBH modules

The original four modules being developed by the OECD for the ESO are derived from PIAAC modules of the same name which were field trialed and administered in the main PIAAC 2011-12 data collection. The Career Interests and Intentionality (CII) module will provide information about the employment status and vocational interests of students in the PISA Young Adult Follow-up Study, at approximately age 18, which can be related to their performance on PISA at age 15. Questions about employment status will provide detailed information about what students are doing for work and/or if they are seeking employment, which will be especially useful for understanding the current experiences of students who did not go on to higher education. For all students in the sample the module will provide information about occupational interests and intentions that can be related to performance on PISA.

The Behavioral Performance Competencies (BPC) module addresses a range of non-cognitive constructs, including well-being, self-control, cooperation, and responsibility. This module will provide a way to relate non-cognitive outcomes of students in early adulthood to earlier performance in reading, mathematics, and science, to address the question of how performance on PISA relates to successful participation in adult life.

The subjective well-being and health (SWBH) module examines the main components of subjective well-being: life evaluation and positive and negative affect in addition to subjective health and objective health indicators in line with the measures described in the World Health Organization’s agenda. Research has shown a relationship between health and well-being for education and work-related outcomes, with healthy individuals being more productive and more successful in their personal and professional lives. In the PISA Validation Study this information will be examined in relation to the previous performance on the PISA assessment, with education and employment status taken into account. Collecting the information on subjective well-being and health could, for example, answer a question whether young adults’ life satisfaction has an association with how well they scored on the reading assessment in PISA or on the literacy assessment as an adult; or, show a relationship between education and employment outcomes and sleeping, eating, and exercise habits.

As noted, the CII, BPC, and SWBH modules have already been used in the field for PIAAC 2011-12 data collection so their extension to ESO does not require further testing.

Rationale for additional non-cognitive survey items

As indicated above, NCES proposes to augment the ESO questionnaire modules with 16 additional items as an ‘add-on’ module that, from a respondent perspective, will seamlessly blend with the ESO. The rationale for proposing these additional items is provided below. These additional items are also taken from the PIAAC 2011-12 questionnaires but were not included in the ESO modules distributed by the OECD. NCES believes these additional items will bring added analytic insight to the results of the PISA Validation Study.

It is estimated that over 80 percent of the U.S. 15-17 years-old cohort go on to some form of postsecondary education (ELS: 2002-2012)². Moreover, some proportion of these young adults stays back in high school. Therefore, four questions on Current education study status (participation; level of degree; area of study) are proposed to be added. Given that some respondents may have started and stopped their education, or dropped out, within the last year, one question on Studies in the last 12 months is proposed to be added. Additionally, a portion of the respondents may be participating in some form of GED or basic skills courses, leading to three additional questions on Basic education courses in the last 12 months (reading/writing/math, GED, some other adult education) proposed to be added. A small proportion of young adults reports participating in apprenticeship programs. Therefore one question on the Formal apprenticeship in the last 12 months is proposed to be added. NCES is proposing to add these nine questions to allow for a more comprehensive tracking of the PISA-PIAAC Follow-up sample into various forms of formal education.

In addition to the nine questions above, another seven questions are being proposed to address the non-formal educational choices of young adults. While a large proportion of young adults participate in formal education, PIAAC data shows that 55 percent of U.S. 16-19 year olds also participated in non-formal education activities (PIAAC:2012)³. To ascertain whether respondents are participating in non-formal education, five questions on Non-formal learning activities (distance education, on-the-job courses, seminars, other private lessons) are proposed to be included. Finally, with the intent of tracking barriers to non-formal learning activities the respondents may experience, two questions on More/any learning activities, wanted, but could not participate in and reasons for non-participation are proposed to be included.

As indicated earlier, respondents will be requested to complete both the cognitive and non-cognitive instruments online. Appendix B includes the ESO non-cognitive survey items in the modules, including the additional items, proposed to be administered for this study.

A.2 Purposes and Uses of Data

PISA is intended to be an assessment at or near the end of compulsory education of how well students are prepared for further education or entry into the workforce and to contribute to society as functioning young adults. The PISA subject matter assessments are not developed by canvassing the curricula offered in participating countries, as is done in other large-scale international assessments. The PISA assessment frameworks and items are developed by conferring with subject matter experts around the world and representatives of participating countries about the mathematics, reading, and science competencies required for 15-year-olds to successfully transition to further education, the workforce, and otherwise as functional contributing members of society. For 2012, the mathematics framework was also circulated internationally among representatives of employers and postsecondary institutions for their comments on the appropriateness of aspects of the PISA mathematics framework.

Because of its emphasis on assessing what is thought to be important for later success rather than assessing what has been taught, PISA must turn to sources of evidence other than participating countries’ curricula in order to check that it is assessing what it intends to assess. Currently, the main source for PISA for this sort of validity checking is in the form of experts drawn from across participating countries, as described earlier. Also, two other countries have performed longitudinal studies based on PISA. A longitudinal study in Canada has followed 30,000 students who participated in PISA 2000, gathering information every two years on their educational experiences, transitions, and attainment and employment outcomes (OECD 2010). A similar study drawing on a smaller sample was conducted in Denmark (Schleicher 2007).

What is lacking is an empirical linkage between PISA and the sorts of skills and outcomes required for successful participation in adult life. The proposed study will provide data on how performance on PISA, at age 15, relates to literacy, numeracy, and problem-solving skills; educational attainment and education and workforces experiences; skills use in daily life; and other non-cognitive outcomes at approximately age 18. For example, what thresholds of performance on PISA are associated the ability to successfully navigate the literacy, numeracy, and problem-solving demands of daily life in the adult world? What thresholds are associated with high school graduation and successful transition to higher education or entry-level workforce? How does performance on PISA relate to measures of subjective well-being at age 18, such as life satisfaction and healthy habits?

By linking performance on PISA to subsequent outcomes in young adulthood, the study will provide the sort of validity evidence needed to understand how well PISA is measuring the skills needed for successful participation in adult life, as it purports to do. The study will also strengthen our understanding of U.S. performance on PISA and its implications for U.S. college and career readiness and for the skills of our future workforce. This information can, in turn, also be used to inform the further development of PISA and future surveys of adult skills.

A.3 Use of Information Technology

Study instruments will be administered through the Internet. Respondents will be invited to log on to a secure website to take the assessment and questionnaire. The use of web-based instruments reduces the burden required for data collection, scoring, and data processing.

A.4 Efforts to Identify Duplication

Comparable U.S. data are not available from other data sources; nor would a PISA follow-up study make any data redundant. The purpose of this study is to look at the relationship of performance on PISA to performance on similar domains and to other outcomes after a period of time, to validate PISA’s measures. No other study in the United States has collected follow-up data from or about PISA students so this would be a unique source of data to accomplish this objective. The current study could produce some information that is similar to a subset of information from the High School Longitudinal Study (HSLS), which follows a similar age group. However, HSLS does not provide data that allow for the validation of PISA measures or that are directly comparable with data from other countries.

A.5 Minimizing Burden for Small Entities

This study will not impose burden on small businesses or small entities.

A.6 Frequency of Data Collection

Currently the main data collection is planned for winter 2015/2016. A field trial of procedures will take place in September 2015. Although there are currently no plans for a second follow-up study with the PISA 2012 students, NCES may decide at a later date to re-contact the cohort sample to collect additional data at another point in time on education and workforce experiences and skills in order to evaluate PISA measures against longer term outcomes.

A.7 Special Circumstances of Data Collection

No special circumstances of data collection are anticipated.

A.8 Consultations outside NCES

NCES has sought input from a wide range of individuals with knowledge of PISA and PIAAC. Both PISA and PIAAC are sponsored by the OECD and are coordinated internationally by a consortium of contractors responsible for various aspects of the studies. Key persons from the OECD and key contracting organizations involved in the international design, development, and operation of PISA and PIAAC are as follows.

Organization for Economic Cooperation and Development

Andreas Schleicher, Indicators and Analysis Division

2, rue André Pascal, 75775 Paris Cedex16, FRANCE

William Thorn, PIAAC

2, rue André Pascal, 75775 Paris Cedex16, FRANCE

Educational Testing Service

Irwin Kirsch, Project Director, ETS Corporate Headquarters

660 Rosedale Road, Princeton, NJ 08541 USA

A.9 Provision of Payments or Gifts to Respondents

As part of the planned efforts to reach response rate goals, NCES proposed giving sampled respondents the same incentive used for the PIAAC Main Study: a $50 cash card to thank the respondent for their time and effort spent taking the assessment and answering the survey. The amount of burden is equivalent to that of the PIAAC respondents. NCES received OMB approval to offer a $50 incentive for the PIAAC Main Study after the results of an incentive experiment during the 2010 PIAAC Field Test indicated that $50 improved response rates compared with lower amounts (OMB# 1850-0870). The $50 incentive was approved for this study in the Initial Contact and Address Updates request (OMB# 1850-0900 v.1).

Due to the unclustered sample, it would be prohibitively expensive to conduct an in-person data collection. A web approach has been designed to administer the ESO to PISA 2012 students. During recruitment, respondents will be supplied with an unloaded cash card that will be loaded upon their completion of the ESO assessment and questionnaire. The card can be used by the respondent anywhere a MasterCard is accepted, once remuneration has been added to it. Until the survey is completed, the card will act as a visual incentive to the respondent to complete the survey.

A.10 Assurance of Confidentiality

Procedures for handling confidential aspects of the study that will be used in the PISA Validation Study will mirror those used in past administrations of PISA. Expertise in data security and confidentiality was a significant criterion in the selection of the PISA contractor. The plan for maintaining confidentiality includes signed confidentiality agreements and notarized nondisclosure affidavits obtained from all personnel who will have access to individual identifiers; personnel training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; controlled and protected access to computer files under the control of a single data base manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and operator-manned in-house computing facility.

The physical and/or electronic transfer of personally identifiable information (particularly names and addresses) will be limited to the extent necessary to perform project requirements. This limitation includes both internal transfers (e.g., transfer of information between agents of Westat) and external transfers (e.g., transfers between Westat and NCES, or between Westat and another government agency assisting in data collection). All data files constructed to conduct the study will be maintained in secure network areas at Westat. These files will be subject to Westat’s regularly scheduled backup process. Backups are stored in secure facilities on site as well as off site. These data are stored and maintained in secure network and database locations where access is limited to those Westat staff who have specifically authorized access. Access is only granted once a staff member is assigned to the project and has completed the NCES Affidavit of Non-disclosure. Identifiers are maintained in files required to conduct survey operations that are physically separate from other research data and that are accessible only to sworn agency and contractor personnel.

The PISA Validation study will follow all applicable federal laws pertaining to security and confidentiality of personally identifiable information (PII). The laws pertaining to the collection and use of PII are clearly communicated in correspondence with participants, per NCES requirements (see materials in Appendix A). Materials will carry a statement addressing confidentiality as follows:

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C., § 9543). Your participation is voluntary and the information you provide may be used only for statistical purposes and may not be disclosed or used, in identifiable form for any other purpose except as required by law (ESRA 2002, 20 U.S.C., § 9573). Individual responses will be combined with those from other participants to produce summary statistics and reports.

Westat will deliver data files, accompanying software, and documentation to NCES at the end of the main study. Neither respondents’ names nor addresses will be included on any data file. Furthermore, prior to the release of any data file, NCES will conduct a disclosure risk analysis, per NCES standard protocol.

A. 11 Sensitive Questions

The survey does not include questions usually considered to be of a highly sensitive nature, such as items concerning religion, substance abuse, or sexual activity.

A. 12 Estimates of Response Burden

Table A-1 shows the estimated respondent burden for field trial and main study recruitment and data collection. There are combined total burden hours of 2,347 hours for the field test and main study data phases. These include communications with respondents asking them to formally participate in the survey, completion of the non-cognitive questionnaire, and completion of the cognitive assessment. At an estimated $7.25 per hour (federal minimum wage), the estimated cost to respondents for the total burden time associated with the address updates (2,347 hours) is $17,016.

Table A-1. Estimated burden to respondents*

Activity	Sample size	Estimated response rate	Number of Respondents	Number of Responses	Estimated average burden (minutes)	Total burden (hours)
Address update requests - main study sample (5)	4,729	1.00	4,729	4,729	5 min per request 25 min total	1,970
Address update requests – field trial sample (3)	1,081	1.00	1,081	1,081	5 min per request 15 min total	270
Total burden address updates			5,810	5,810		2,240
Field trial
Communications	235	0.85	200	200	10	33
Questionnaire	235	0.85	200	200	35	117
Assessment	235	0.85	200	200	90	300
Data collection
Communications	3,447	0.85	2,930	2,930	10	488
Questionnaire	3,447	0.85	2,930	2,930	35	1,709
Assessment	3,447	0.85	2930	2,930	90	4,395
Total burden - data collections (communications and questionnaire)			3,130	6,260		2,347

*Total burden in the current request is for the field trial and main study communications and data collection of the questionnaire. Gray shaded rows indicate the portion of burden that was previously approved and completed, and entries in gray font represent burden associated with assessments, which is not subject to PRA approval and therefore not included in the burden totals.

A.13 Estimates of Cost to Respondents

Other than the burden time associated with completing the address updates and study instruments, the study imposes no additional cost to respondents.

A.14 Cost to Federal Government

The cost to the Federal Government for conducting the validation study is estimated to be $1,945,602 over a four-year period, based on the cost for the Validation Study task in the PISA 2015 contract (ED-IES-13-C-0006). This includes tracing and tracking sample respondents, data collection, use of the OECD’s ESO instruments, and analysis and reporting. These figures include all direct and indirect costs of the project.

A.15 Program Changes or Adjustments

The increase in respondent burden for this collection is due to the addition of the SWBH module to the study questionnaire per OECD requirement.

A.16 Publication Plans and Timetable

NCES will release a general audience report and technical/operations report, along with restricted-use data files. Electronic versions of each report will be made available on the NCES website. The expected data collection dates and a tentative reporting schedule are provided below.

Date	Activity
Fall 2012	Collect contact information from PISA 2012 students (completed)
Fall 2013 – September 2015	Establish initial contact and conduct semi-annual contact with respondents for address updates (ongoing)
September 2015	Conduct field trial of procedures
Winter 2015/2016	Web-based main study data collection
2016-2017	Reporting: general audience report; survey operations report; restricted-use data-files

A.17 Display OMB Expiration Date

The OMB expiration date will be displayed on all data collection materials.

A.18 Exceptions to Certification Statement

No exceptions are requested to the "Certification for Paperwork Reduction Act Submissions" of OMB Form 83-I.

References

Goodman, M., Finnegan, R., Mohadjer, L., Krenzke, T., and Hogan, J. (2013). Literacy, Numeracy, and Problem Solving in Technology-Rich Environments Among U.S. Adults: Results from the Program for the International Assessment of Adult Competencies 2012: First Look (NCES 2014-008). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved [date] from http://nces.ed.gov/pubsearch.

Organization for Economic Cooperation and Development and Statistics Canada. (1997). Literacy Skills for the Knowledge Society: Further Results from the International Adult Literacy Survey. Paris: OECD.

Organization for Economic Cooperation and Development and Statistics Canada. (2005). Learning a Living: First Results from the Adult Literacy and Life Skills Survey. Paris: OECD.

Organization for Economic Cooperation and Development. (2009). PISA 2009 Assessment Framework - Key Competencies in Reading, Mathematics and Science. Paris: Author.

Organization for Economic Cooperation and Development. (2010). Pathways to Success: How Knowledge and Skills at Age 15 Shape Future Lives in Canada. Paris: Author.

Organization for Economic Cooperation and Development. (2012). Literacy, Numeracy and Problem Solving in Technology-Rich Environments - Framework for the OECD Survey of Adult Skills. Paris: Author.

Schleicher, A. (2007). Can competencies assessed by PISA be considered the fundamental school knowledge 15-year-olds should possess? Journal of Educational Change, 8: 349-357 (published online July 24, 2007).

1 The sample of ~5,200 students took the PISA mathematics, science, and reading literacy assessment. PISA 2012 also included an assessment of financial literacy given to an additional ~1,100 students. In order to focus the Validation Study on the main PISA domains, the proposed study does not include the students who took financial literacy except in the field trial.

2 Lauff, E., and Ingels, S.J. (2013). Education Longitudinal Study of 2002 (ELS:2002): A First Look at 2002 High School Sophomores 10 Years Later (NCES 2014-363). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved [11/11/2014] at http://nces.ed.gov/pubs2014/2014363.pdf

3 Organization for Economic Cooperation and Development (OECD), Program for the International Assessment of Adult Competencies (PIAAC), 2012. Retrieved from PIAAC IDE [11/11/2014].

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	PREFACE
Author	Janice Bratcher
File Modified	0000-00-00
File Created	2021-01-24