Response to OMB passback

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: November 17, 2010

TO: Shelly Martinez, OMB

FROM: Dana Kelly, NCES

THROUGH: Kashka Kubzdela, NCES

RE: Response to 10-07-2010 through 11-17-2010 OMB Passbacks for PISA 2012 Recruitment and Field Test (OMB# 1850-0755 v.10)

The revisions referred to in the responses below have been made in Parts A and B.

Q1: Which countries are doing paper only and which are doing computer?

Response: The table below shows all jurisdictions participating in PISA 2012 and indicates which countries are doing the optional components in which the United States is planning to participate. The table is sorted by participation in computer-based assessment of reading and mathematics, then computer-based problem-solving, then paper-pencil financial literacy.

All 67 jurisdictions are administering the PISA paper-and-pencil assessment of mathematics, science, and reading literacy. This is a core component of PISA 2012 and required of all participating countries. The computer-based assessment of problem-solving is also a core component of PISA 2012, although some countries are unable to administer it (45 jurisdictions will do). The computer-based reading and mathematics assessments are optional components of PISA 2012 (28 jurisdictions will do). The computer-based assessments are separate and distinct from the paper-based assessments of reading and mathematics. Data from the paper-based and computer-based versions will be scaled and reported separately from each other. No country is doing only computer-based assessments, and 19 jurisdictions will conduct paper-pencil financial literacy.

Participation in PISA 2012 Assessments, by Jurisdiction

Jurisdiction	Paper and Pencil Assessment of Math, Science, and Reading	Computer-based Assessment of Problem solving	Computer-based Assessment of Literacy (Math & Reading)	Financial Literacy (paper-and pencil)
Australia	Yes	Yes	Yes	Yes
Belgium	Yes	Yes	Yes	Yes
Brazil	Yes	Yes	Yes	Yes
China (Shanghai)	Yes	Yes	Yes	Yes
Colombia	Yes	Yes	Yes	Yes
Estonia	Yes	Yes	Yes	Yes
Israel	Yes	Yes	Yes	Yes
Italy	Yes	Yes	Yes	Yes
Poland	Yes	Yes	Yes	Yes
Slovak Republic	Yes	Yes	Yes	Yes
Slovenia	Yes	Yes	Yes	Yes
Spain	Yes	Yes	Yes	Yes
United States of America	Yes	Yes	Yes	Yes
Chile	Yes	Yes	Yes	No
Chinese Taipei	Yes	Yes	Yes	No
Denmark	Yes	Yes	Yes	No
Hong Kong-China	Yes	Yes	Yes	No
Hungary	Yes	Yes	Yes	No
Ireland	Yes	Yes	Yes	No
Japan	Yes	Yes	Yes	No
Macao-China	Yes	Yes	Yes	No
Norway	Yes	Yes	Yes	No
Portugal	Yes	Yes	Yes	No
Republic of Korea	Yes	Yes	Yes	No
Russian Federation	Yes	Yes	Yes	No
Singapore	Yes	Yes	Yes	No
Sweden	Yes	Yes	Yes	No
United Arab Emirates	Yes	Yes	Yes	No
Croatia	Yes	Yes	No	Yes
Czech Republic	Yes	Yes	No	Yes
France	Yes	Yes	No	Yes
Argentina	Yes	Yes	No	No
Austria	Yes	Yes	No	No
Bulgaria	Yes	Yes	No	No
Canada	Yes	Yes	No	No
Costa Rica	Yes	Yes	No	No
Finland	Yes	Yes	No	No
Germany	Yes	Yes	No	No
Luxembourg	Yes	Yes	No	No
Malaysia	Yes	Yes	No	No
Montenegro	Yes	Yes	No	No
Netherlands	Yes	Yes	No	No
Serbia	Yes	Yes	No	No
Turkey	Yes	Yes	No	No
Uruguay	Yes	Yes	No	No
Albania	Yes	No	No	Yes
Latvia	Yes	No	No	Yes
New Zealand	Yes	No	No	Yes
Cyprus	Yes	No	No	No
Georgia	Yes	No	No	No
Greece	Yes	No	No	No
Iceland	Yes	No	No	No
Indonesia	Yes	No	No	No
Jordan	Yes	No	No	No
Kazakhstan	Yes	No	No	No
Liechtenstein	Yes	No	No	No
Lithuania	Yes	No	No	No
Mexico	Yes	No	No	No
Peru	Yes	No	No	No
Qatar	Yes	No	No	No
Romania	Yes	No	No	No
Switzerland	Yes	No	No	No
Thailand	Yes	No	No	No
Trinidad and Tobago	Yes	No	No	No
Tunisia	Yes	No	No	No
United Kingdom	Yes	No	No	No
Viet Nam	Yes	No	No	No
Total Yes	67	45	28	19
Total No	0	22	39	48

Q2. With which countries would we not be able to be compared if the U.S. decided to pursue the more limited data collection options for the full scale study?

Response: As stated in the response to Question 1, all participating jurisdictions (show in the above table) are administering the PISA paper-and-pencil assessment of mathematics, science, and reading literacy. If the United States elected to only administer the core components (paper-and-pencil assessment mathematics, science, and reading and computer-based problem solving) comparisons can be made with all 67 participants in mathematics, science, and reading and with 45 jurisdictions in problem solving.

Q3. What are the specific criteria (e.g., at a certain level of increased cost) ED will use to decide among the different options?

Response: It is our intention and expectation that the United States will administer in the main survey the paper-and pencil reading, mathematics, and science assessments, the computer-based problem solving assessment, the computer-based reading and mathematics assessments, and the paper-and-pencil financial literacy assessment. However, in our request for OMB clearance we leave open the possibility that the United States may opt for a more limited data collection depending on our experience in the field trial.

The PISA 2012 field trial will be the first time the United States has implemented a computer-based assessment in PISA. In PISA 2009, 19 countries administered the Electronic Reading Assessment (ERA), but the United States did not. In 2006, three countries administered the Computer-based Assessment of Science (CBAS), but the United States did not. So, while computer-based assessments have been administered in PISA in the past two rounds, the United States did not participate in these previous computer-based activities. NCES has been very cautious in moving to the computer-based model by carefully considering the costs, operational feasibility, and burden on schools and students. However, the plan for PISA internationally is to move toward a fully computer-based mode of data collection in 2015. It is thus very important that the United States administer the computer-based assessments in PISA 2012 in order to fully understand the ramifications and outcomes associated with computer-based assessments as a data collection mode. In addition, the reading and mathematics computer-based assessments in PISA 2012 will be assessing elements of the assessment frameworks not optimally assessed through paper-based assessments and thus we expect to learn additional information about students’ skills in these areas, beyond what we learn through the paper and pencil assessments. At the same time, it is important to understand that the computer-based reading and mathematics assessments are not full assessments of those domains and are not equivalent to the paper-based versions of the assessments. The computer-based and paper-based versions will be scaled and reported separately and will include only a subset of all PISA countries, shown in the response to Question 1.

The evaluation of which configuration of PISA to employ in the main study will be based on several factors taken together: data quality, ability to achieve acceptable response rates, and cost. With respect to data quality, we will evaluate the quality of the assessments based on the field trial item statistics, we have every reason to expect that the assessment items will function properly. We have reviewed all items developed for the field trial and cognitive labs and limited trials conducted by the international consortium (in other countries) suggest that the items will work well. We are primarily concerned with whether we can achieve acceptable response rates under a design in which in each school all sampled students take the PBA and a subsample also take the CBA. Our main focus in the field trial will be learning whether and how we can do so efficiently and effectively. In addition, we will examine whether, given the experience in the field trial, we believe we will be able to administer the PBA and CBA in the main study within the budget currently available for the main study data collection. If we believe that a successful administration would require us to spend in excess of, say, 10-15% of the budget currently available for recruitment, training, and data collection, and if other information also suggests that the CBA is problematic, we will reconsider administering CBA in the main study. Lastly, administration of the financial literacy assessment in the main study is dependent on NCES obtaining additional funding.

Q4.       How does the burden in the PISA field test schools that are only getting the computer assessment component compare to the other field test schools and the 2009 field test schools?  We are trying to better understand how burden (including assessments) will look to the schools being recruited.

Response: The international procedure is to administer paper-based assessments to all PISA students and subsample students for the CBA; this will be the procedure to be used in the main study should both paper-based and computer-based assessments be implemented in the main study. However, for the field trial, there are sample size requirements driven by the need to obtain sufficient sample size for item analyses. Countries are required to have 1,800 CBA assessed students, and 800 students for the paper-based assessment. For efficiency and to minimize burden in the field trial, we divided the sample into schools in which all students will take the paper-based assessment and a subsample (about half) will take the computer-based (PP+CBA schools) and schools in which all sampled students will take the CBA (CBA-only schools).

For students doing the PB assessment, the time requirement is 3 hours of total time for each sampled student (2 hours for assessment, 30 minutes for the background questionnaire and 30 minutes for breaks, and administration directions). This is the same as was the case in the PISA 2009 field trial and main study. For students taking CBA, the time is approximately 1 hour of total time per student. This includes logging in, directions and demonstration items, and the assessment (2 20-minute parts). Thus, for students taking both paper-based assessment and CBA (those that are subsampled for CBA) the total combined time per student is approximately 4 hours, but students will have to return for the CBA portion of the assessment later the same day.

In all schools, whether PP+CBA or CBA only, all school principals will be asked to complete a school questionnaire and both will have similar levels of burden associated with obtaining student listing forms and locating space for the assessment. Unlike in the 2009 field trial when we did not administer CBA, we will need to be in the school for approximately two additional hours beyond the usual amount of time, to set up the testing space for and conduct the CBA. While we will make every attempt to minimize our time in each school, for schools with two sessions (PB+CB), it may be necessary for data collection staff to be in the school for upwards of 8 hours, depending on when the second (CBA) session is held. This does not represent burden to the school in terms of time they need to expend on PISA, but it represents the time that PISA staff will be in the school, and the fact that assessment space at the school will be occupied for a longer time by PISA. Text in section A-9 was revised to better explain the time we expect to spend in schools.

Q5. Related, how will NCES interpret incentive experiment results given that schools will be randomly assigned but within the treatment and control groups there are two distinct burden levels being imposed both on schools and on students?

Response: This would seem to be neutral in the sense that there are both PP+CBA schools and CBA-only schools in each group. We will be able to look at 4 sets of schools – PP+CBA with high incentive, PP+CBA with low incentive, CBA-only with low incentive, CBA-only with high incentive. However, our real interest is the PP+CBA groups. As mentioned in #4 above, schools in which students take only CBA have been added to meet the international requirement of 1,800 CBA-assessed students; we do not intend or expect to have that scenario in the main study.

Q6. Is the experiment literally proposing to offer one amount of money to students for the first session and a separate amount of money for the second?  Or is it a set amount for doing both?  What is the rationale either way?

Response: We will offer a set amount to the student depending on what they were sampled for. In PB+CBA schools students sampled for the PB-only will be offered $20/$40 (depending on whether the school is in the Incentive 1 group or Incentive 2 group – see table A-2 in Part A) and students sampled for both the PB and CBA will be offered $35/$60 (again, depending on which incentive group applies). The rationale for offering more to students taking both PB and CBA is that those students are doing more, will experience a significantly more disrupted school day, and should be remunerated for the additional inconvenience and effort. In CB-only schools, we will offer $15/$20 to students to take the CB assessment. The text in section A-9 of Part A has been revised to clarify this arrangement.

Q7.   Is the payment proposed to be directly to the children or to their parents?  What form is the payment proposed to take?  How did NCES issue payments in 2009?  Were parents involved in any way?

Response: The payments are proposed to be to the students, as they were in PISA 2003, 2006 and 2009. However, the parents were informed of the amount of the payment the students will receive in the consent form/letter in advance of the assessment. The payments will likely take the form of a personal check. There are advantages and disadvantages to payment by check. This was the method used in the field test and main study in 2009 and it worked well on the whole. Checks allow a great degree of control and tracking ability of incentives in the field, are easy to distribute and have less risk than other forms, such as cash. For example, we cancelled the checks not deposited in PISA2009 after certain period of time, so we were able to control the money effectively. There are some instances where students are not familiar with checks or do not have a bank account. We have been able to rectify these situations by giving them information and help in cashing their checks.

Parents were informed and/or gave their permission (either implied or explicit) for their children to participate in PISA. We did not supply payments to parents. Parents are able to contact the project directly via phone or email to obtain information or voice concerns.

The text in section A-9 of Part A has been revised to clarify the recipient and form of the student incentive.

Q8. We would like to see the control for the experiment be the same amount as was offered in 2009 rather than a higher amount.

Response: Table A-2 shows the amounts. The amounts shown for the Incentive Group 1 (the control group) were the amounts offered in 2009. The amounts for CBA were added, as in 2009 there was no CBA. A footnote has been added to Table A-2 to clarify this.

Q9. We do not see a justification for increasing the nonschool hour payment above $50.  We are concerned that these payment levels are probably likely than anything else we have approved for children in a variety of contexts.

Response: We try very hard to limit the number of assessments that are given out-of-hours. In 2009, only 5 percent of schools were assessed outside of regular school hours and we intend to minimize them in the PISA 2012 as well. The increased incentive over and above the “normal” incentive compensates students for travel time and other activities (work, sports) that a student may miss to participation in the assessment out of hours. In PISA 2006 and 2009, we originally received approval from OMB to offer $35 for out-of-hours assessments. After the original approval, we received OMB approval to offer $50 for after-school and $75 for Saturday assessments. In 2006 we requested and receive approval to increase the amounts from $35 to $50-75 in the middle of data collection in order to achieve acceptable response rates. In 2009 we requested and received before the start of the study an approval for these increased amounts (as a change request), although ultimately it was not necessary to offer the increased amounts for out-of-hours assessments (that is, we were able to achieve participation with a $35 given to students taking the assessment out of normal school hours). In PISA 2012, we would begin by offering $35 to students taking the assessment out of normal school hours but are requesting permission to offer the increased amounts if it becomes necessary to do so in the main study in order to achieve acceptable response rates. The text in section A-9 of Part A has been revised to clarify our request and how the out of hours assessment incentive worked in 2006 and 2009.

Q10. Are the school coordinators proposed to receive the higher amount of incentive those in the higher burden schools, the schools assigned randomly to receive the higher school amount, or via some other approach?  How will the results be analyzed given the interaction.

Response: The incentive experiment is an attempt to learn how different incentive amounts affect respondents’ participation in PISA. This has not been previously examined in any real way in PISA in the US. The incentive experiment is only one of a number of ways in which recruitment strategies will be evaluated for the main study so as to achieve the required response rates. Other sources of feedback for determining the main study recruitment strategy include the recently approved focus groups with principals and students, the field trial recruitment experience as based on the reports from the field, and the planned debriefings to be held with principals and school coordinators of responding and non-responding field trial schools. In the field test, the assignment of the school coordinators as well students to the two different treatment levels will be based on the incentive treatment their school is assigned to (high or low).

Q11. Please confirm the cost information in A14 – ie, the one year and three year costs.

Response: This OMB package was prepared prior to the PISA2012 contract being awarded and thus the figures in the original submission were estimates. The contract was awarded in August and the following figures are based on the contract budget.

The cost to the Federal Government for conducting the PISA 2012 field trial is estimated to be $2,576,448 over a 1-year period. The total cost to the Federal Government for conducting the PISA 2012 main study is estimated to be $1,136,329 per year for a 3-year period for a total of $3,408,987 for the main study and $5,985,435 altogether. These figures include all direct and indirect costs of the project, and are based on the United States administering the core assessments (computer-based problem solving and paper and pencil mathematics, science, and reading) and optional computer-based mathematics and reading assessments. In addition to these costs, financial literacy (not included in the total cost of the $5,985,435) is estimated to cost $383,158 for the main study. Thus, the total cost to the government, should all core and optional components be included in the field trial and main study, is $6,368,593.

The text in section A-14 has been revised to reflect these updates.

Q12.   For the main study, will any effort be made to unduplicate sample with NAEP or other NCES studies?  If not, why?

Response: We will make every effort to avoid overlap with grade 12 NAEP, which will be conducted in spring 2013. However, the NAEP 2013 sample will not be drawn until April 2012 so we will work with the NAEP program to request that the NAEP sample does not include PISA schools. NCES’ High School Longitudinal Survey (HSLS) will also be in the field in spring 2012, collecting data from the HSLS: 2009 grade 9 cohort. We will avoid overlap with these schools, as we did in the PISA 2009 data collection.

Q13.   OMB remains concerned about the proposed high level of payment to children.  We will approve an incentive experiment where the “control” group receives a total of $20 (not $20 plus $15).  We will approve a treatment level of $40, but no higher.

Response: We are very concerned about the PISA incentives and wanted to explain our perspective as well as better understand OMB's.

In 2008, PISA 2009 was approved to give $20 for paper and pencil plus $15 for computer-based assessment.  We ended up not doing the computer-based assessment, but the amounts we are asking for this year are no different from those OMB approved the last time.

To explain our concerns: we have never asked students to come back the same day for a second assessment session and need to make sure that we understand what it takes to assure that they come back.  In PISA 2012 it is very important that we administer the computer-based assessment, to assure that in 2015, when, per OECD’s plan, PISA will be exclusively computer-based, we will have worked out any potential major issues and understand the realities of administering PISA as a computer based assessment.

In the field test, in about half of the schools, half of the students will be asked to participate in two testing sessions. All students will be getting $20 (as in 2009) for 2 hrs of paper and pencil, and it would likely be seen as unfair, by students and schools, if students in the same school, who are asked to do the 2 hrs of paper and pencil session plus come back later that day, probably after lunch, for another 1 hr assessment session, would also receive only $20.  Giving all students the same amount, despite a significant increase in burden to half of them, would be difficult to explain and we might find ourselves with students not attending the second session.  In many schools students have to make up schoolwork conducted during missed classes on their own time.  Even though we will be administering only the computer-based assessment in some schools during the field test (in order to have enough data to evaluate the items), we will exclusively administer the two-session version during the main study, to minimize cost and to facilitate analyses on the relationship between paper-and-pencil and computer-based measures.  Recruiting more schools and students to avoid double sessions for students would be far more expensive than the second session incentive, it would mean a loss in what we learn analytically, and in fact it would not comply with the PISA international procedures.

The field test is our only opportunity to learn whether and how we can obtain the necessary response rates, and we are hoping to approach school and students with a request that seems fair in relation to the burden imposed.

We are asking to conduct the incentive experiment, as well as the already approved focus groups, because school and student participation rates have been declining each time and we are concerned about response rates this year both because of the decline we have already observed and because we don't know what the reaction of schools and students will be to asking them to do two assessment sessions (as we will ask in the main study).  We have thought hard about the incentive levels we are proposing for The PISA field test, and we would like to revise the proposed amounts for the experimental group to:

• Students taking only paper-and-pencil session: $30

• Students taking paper-and-pencil session AND computer-based session: $30 + $20

• Students taking only computer-based session: $20

This would revise the proposed experiment as follows:

Recipient	Incentive 1 (2009 amounts)*	Incentive 2
Schools	$200*	$800
School coordinators	$100*	$200
Students
P&P only	$20*	$30
P&P and CBA	$20* + $15 for CBA	$30 + $20 for CBA
CBA only	$15	$20

P&P = paper and pencil assessment; CBA = computer-based assessment.

*These amounts were offered to schools, school coordinators and students taking the P&P in 2009; however, CBA was not included in 2009 and thus the amount for CBA in the proposed experiment, although approved by OMB for 2009, was not implemented in 2009.

 

If these incentives are problematic from OMB's point of view, we would respectfully request a chance to discuss the issues in order to come up with a solution that would allow us to solidly explore how to successfully conduct dual PISA sessions in schools in this transition from PISA paper and pencil only to computer-based.

Q14.   We also will ask NCES to report back to OMB prior to making a final decision on whether to implement the computer based option in the full scale, as we are concerned both about the recruiting/response rates and cost.

Response: NCES will make sure to keep OMB informed about the field test results and full scale study plans. We will also be happy to discuss the computer based option during the potential meeting on the incentives (see response to Q13 above).

Q15.   We’d also like to have, for the record, a justification for the parental employment questions, including why it is important to ask about mother and father separately in a gender specific way (as opposed to something my gender neutral like parent 1 and parent 2) as well as an assessment of the utility of the full set of questions, given widely acknowledged difficulty in accurately querying youth about such details.

Response: Response: Items about parental employment are international questions that have been administered in this fashion since the beginning of PISA in 2000. The employment questions are asked as two open-ended questions - job title and main job duties. The responses are taken together and coded to the International Standard Classification of Occupations (ISCO) and then mapped to an international socio-economic index of occupational status. PISA 2012 will use ISCO-08 (http://www.ilo.org/public/english/bureau/stat/isco/isco08/index.htm). The PISA international consortium undertook validities studies in several countries to study the question of how well 15-year-olds could be expected to report the occupations of their parents (see the PISA 2000 Technical Report at http://www.pisa.oecd.org/dataoecd/53/19/33688233.pdf). Generally, these studies indicate that useful data on parental occupation can be collected from 15-year-old students.

In addition to creating a simple index of highest occupational status of parents, parental occupational information is combined with information about parental education and household possessions to create the PISA index of economic, social and cultural status.

We have pasted below the instructions to and items from the section of the proposed PISA 2012 field trial version of the student questionnaire that relate to family composition, parents’ education and parents’ employment status. Included are notes for the National Project Managers and translators and notes germane to the production of the test administrator guide (these will not appear in what is presented to students). These are the international versions that the United States will adapt to the U.S. context.

Students are instructed at the beginning of the section on family and home that mother and father refer to “your mother and father or those persons who are like a mother or father to you — for example, guardians, step‑parents, foster parents, etc.” NCES can propose to the international consortium that we remind students that mother and father include persons who are like a mother and a father such as guardians, step-parents, and foster parents in each question about parents’ education and employment. We can also include these reminders in the test administrator manual to advise test administrators on how to respond to questions students might ask about this issue. Changing the items so that generic parent labels are used would probably not be accepted by the international consortium as it would make the U.S. data not comparable to other countries and would affect the ability to measure trends.

Q16.   On PISA, I wanted to let you know that I’ve done some additional consulting here and we remain committed to offering no more than $40 per child.  We would be pressed to see ourselves approving over $40 for even an out of-school activity in the many other contexts in which we are asked to consider incentives for teens.  I’m sure if you consider our conversations on focus groups and other activities for teens you will recall this hesitation.  Therefore, since this study is conducted during the school day, we cannot see ourselves agreeing to a level even higher.   We are willing to let NCES identify the 2 or 4 levels that you want to use during the field test.  We would appreciate you letting us know the levels you plan to propose before resubmitting. 

Response: We will offer all students a fixed amount: $25 to those chosen to do only paper-and-pencil assessment, $15 to those chosen to do only the computer based assessment (CBA), and $40 to those chosen to do both. Therefore students will not be part of the proposed experiment, which we have revised as follows:

Recipient	Incentive 1 (2009 amounts)*	Incentive 2
Schools	$200*	$800
School coordinators	$100*	$200
Students
P&P only	$25	$25
P&P and CBA	$25 + $15 for CBA	$25 + $15 for CBA
CBA only	$15	$15

P&P = paper and pencil assessment; CBA = computer-based assessment.

*These amounts were offered to schools, school coordinators and students taking the P&P in 2009.

SECTION 2: ABOUT YOUR FAMILY AND HOME

In this section you will be asked some questions about your family and your home.

Some of the following questions are about your mother and father or those persons who are like a mother or father to you — for example, guardians, step‑parents, foster parents, etc.

If you share your time with more than one set of parents or guardians, please answer the following questions for those parents/guardians you spend the most time with.

Q8	Who usually lives at <home> with you?
	(Please tick one box in each row.)
		Yes	No
a)	Mother (including stepmother or foster mother)	1	2
b)	Father (including stepfather or foster father)	1	2
c)	Brother(s) (including stepbrothers)	1	2
d)	Sister(s) (including stepsisters)	1	2
e)	Grandparent(s)	1	2
f)	Others (e.g. cousin)	1	2

Notes for National Project Manager:

This question has been retained from the PISA 2009 Main Survey (StQ08). This question establishes the student’s current caregivers.

Stem: <home> - refers to that place where the student usually resides, not including boarding school. The term used should connote a family or domestic setting.

Adaptations to this question are a requirement. They need to be described in the Questionnaire Adaptation Spreadsheet and agreed between the NPM and the Consortium.

Notes for Translator:

This question was worded identically in the PISA 2009 Main Survey (StQ08).

Q8: Notes for Test Administrator:

If a student belongs to two households then the questions refer to the household in which he or she spends most time. If a student says he or she spends equal time in two households, then he or she may choose either household, but consistently answer ‘Home’ questions for the chosen household.

Q9a	What is your mother’s main job? (e.g. school teacher, kitchen-hand, sales manager)
	(If she is not working now, please tell us her last main job.)
	Please write in the job title.

Notes for National Project Manager:

This question has been retained from the PISA 2009 Main Survey (StQ09a). This item, in combination with StQ09b, establishes the mother of the student’s main occupation.

This variable is used, in combination with the next question, to generate International Standard Classification of Occupations (ISCO) codes. Please remember to use the ISCO-08 version of the codes.

Examples: It is acceptable to change the examples to more nationally relevant jobs. Do not use obscure or lengthy job titles.

Answer line: ‘job title’ - is the common name of the job. Please use an appropriate term.

Notes for Translator:

This question was worded identically in the PISA 2009 Main Survey (StQ09a).

Q9b	What does your mother do in her main job? (e.g. teaches high school students, helps the cook prepare meals in a restaurant, manages a sales team)
	Please use a sentence to describe the kind of work she does or did in that job.

Notes for National Project Manager:

This question has been retained from the PISA 2009 Main Survey (StQ09b). This item, in combination with StQ09a, establishes the mother of the student’s main occupation.

Use examples that match the job titles given in StQ9a.

Notes for Translator:

This question was worded identically in the PISA 2009 Main Survey (StQ09b).

Q9: Notes for Test Administrator:

‘main job’: If the mother has more than one job, her ‘main job’ is the job in which most time is spent, not necessarily the highest earning job.

If the mother is working at home, doing home duties, please instruct the student to describe this as ‘working at home, doing home duties’.

If a student’s mother (or equivalent female guardian) is deceased, then her last job can still be provided – but if this is unknown the question should be left blank.

Encourage students to answer this question. A general description such as ‘works in an office’ is better than nothing written at all.

Q10	What is the <highest level of schooling> completed by your mother?
	If you are not sure which box to choose, please ask the test administrator for help. (Please tick only one box.)
	<ISCED level 3A>	1
	<ISCED level 3B, 3C>	2
	<ISCED level 2>	3
	<ISCED level 1>	4
	She did not complete <ISCED level 1>	5

Notes for National Project Manager:

This question has been retained from the PISA 2009 Main Survey (StQ10). This item establishes the mother of the student’s highest level of education.

Stem: <highest level of schooling> should be adapted to refer to the sections of schooling that correspond to ISCED levels 1 to 3.

The categories need to be specified using country-specific terms that will be understood by the students responding to the question. Each category needs to be mapped to the ISCED classification of educational levels (see Classifying Educational Programmes — Manual for ISCED-97 Implementation in OECD Countries, 1999 Edition, OECD). The following information about ISCED is taken from this manual.

ISCED Level 1: Primary level of education.

ISCED Level 2: Lower secondary level of education.

ISCED Level 3A: Upper secondary level of education with programmes designed to provide direct access to ISCED 5A.

ISCED Level 3B: Upper secondary level of education with programmes designed to provide direct access to ISCED 5B. Level 3B tends to be more practical and has a vocational orientation.

ISCED Level 3C: Upper secondary level of education with programmes designed to provide direct access to the labour market.

In some countries there may not be an administrative or structural boundary between ISCED 2 and 3 in the educational system. In these cases one may ask about completion of the grade/year level that can be defined as an implicit boundary between ISCED level 2 and 3.

Adaptations to this question are a requirement. They need to be described in the Questionnaire Adaptation Spreadsheet and agreed between the NPM and the Consortium.

Qualifications obtained abroad - increasingly, students have parents whose qualifications were obtained abroad and these may not match the nationally specific categories listed in the questionnaire. The student has been instructed to ask the test administrator if they have any doubt about which option to choose. NPMs should provide some guidelines to Test Administrators in their training on the equivalence of local qualifications to those obtained abroad. The match does not have to be exact. It is more important to try to distinguish between the three general levels: ISCED 5A and above, ISCED 3, and below ISCED 3. The test administrator should ask the student appropriate questions to identify which of the three levels most closely corresponds.

Notes for Translator:

This question was worded identically in the PISA 2009 Main Survey (StQ10). Do NOT use the expressions ‘ISCED 3A’, ‘ISCED 3B’ etc. as the respondent will not understand them.

Q10: Notes for Test Administrator:

This question is asking about completion; that is, obtaining a qualification. Just attending an institution where these qualifications can be obtained is not sufficient.

If a student’s mother (or equivalent female guardian) is deceased, then her qualifications can still be provided – but if this is unknown the question should be left blank.

If the student’s mother obtained her qualifications abroad, please help the student to choose the response from the list that is closest. Asking questions like “How long did she go to school for?” and “Did she go on to university?” should help clarify.

Q11	Does your mother have any of the following qualifications?
	If you are not sure how to answer this question, please ask the test administrator for help. (Please tick one box in each row.)
		Yes	No
a)	<ISCED level 6>	1	2
b)	<ISCED level 5A>	1	2
c)	<ISCED level 5B>	1	2
d)	<ISCED level 4>	1	2

Notes for National Project Manager:

This question has been retained from the PISA 2009 Main Survey (StQ11). This item establishes whether or not the mother of the student has certain educational qualifications.

The categories need to be specified using country-specific terms that will be understood by the students. Each qualification needs to be mapped to the ISCED classification of educational levels (see Classifying Educational Programmes — Manual for ISCED-97 Implementation in OECD Countries, 1999 Edition, OECD). The following information about ISCED is taken from this manual.

ISCED Level 6: Advanced research qualification, devoted to advanced study and original research, requiring submission of a thesis or dissertation of publishable quality.

ISCED Level 5A: Qualification obtained from a tertiary study programme with a strong theoretical foundation typically with a minimum duration of three years’ full time equivalent, providing entry into a profession with high skills requirements or an advanced research programme.

ISCED Level 5B: Qualification obtained in tertiary programmes that are generally more practical/technical/occupationally specific and typically shorter than ISCED 5A programmes. Typically, these programmes have a minimum of two years’ full-time equivalent duration and prepare students to enter a particular occupation.

ISCED Level 4: Qualification obtained in programmes that overlap the boundary between upper-secondary and post-secondary education. They are typically not significantly more advanced than programmes at Level 3 and have a full-time equivalent duration of between 6 months and 2 years.

Adaptations to this question are a requirement. They need to be described in the Questionnaire Adaptation Spreadsheet and agreed between the NPM and the Consortium.

Notes for Translator:

This question was worded identically in the PISA 2009 Main Survey (StQ11).Do NOT use the expressions ‘ISCED 5A’, ‘ISCED 5B’ etc. as the respondent will not understand them.

Q11: Notes for Test Administrator:

This question is asking about completion; that is, obtaining a qualification. Just attending an institution where these qualifications can be obtained is not enough.

If a student’s mother (or equivalent female guardian) is deceased, then her qualifications can still be provided – but if this is unknown the question should be left blank.

If a student has a mother who was trained in a non-university setting but currently has an occupation requiring university education for admission (common examples are teaching, nursing and some accounting occupations) he or she may use response category b).

If the student’s mother obtained her qualifications abroad, please help the student to choose the response from the list that is closest. Asking questions like “Did she go to university?” should help clarify.

Q12	What is your mother currently doing?
	(Please tick only one box.)
	Working full-time <for pay>	1
	Working part-time <for pay>	2
	Not working, but looking for a job	3
	Other (e.g. home duties, retired)	4

Notes for National Project Manager:

This question has been retained from the PISA 2009 Main Survey (StQ12).

This item provides information on the mother of the student’s current employment status; as well as additional information on social background.

Notes for Translator:

This question was worded identically in the PISA 2009 Main Survey (StQ12.

Q12: Notes for Test Administrator:

Students should answer this question with regard to their mother’s main occupation. If e.g. the mother is working part-time but also doing home duties, students should <tick> b) ‘working part-time’.

If a student has no mother (or equivalent female guardian), then he or she should leave this question blank.

Please note that students should tick only one box.

Q13a	What is your father’s main job? (e.g. school teacher, kitchen-hand, sales manager)
	(If he is not working now, please tell us his last main job.)
	Please write in the job title.

Notes for National Project Manager:

This question has been retained from the PISA 2009 Main Survey (StQ13a). This item, in combination with StQ13b, establishes the father of the student’s main occupation.

This variable is used, in combination with the next question, to generate International Standard Classification of Occupations (ISCO) codes. Please remember to use the ISCO-08 version of the codes.

Stem: Examples - it is acceptable to change the examples to more nationally relevant jobs. Please use examples that will help students to place a name on the job title. Do not use obscure occupational titles.

Notes for Translator:

This question was worded identically in the PISA 2009 Main Survey (StQ13a).

Q13b	What does your father do in his main job? (e.g. teaches high school students, helps the cook prepare meals in a restaurant, manages a sales team)
	Please use a sentence to describe the kind of work he does or did in that job.

Notes for National Project Manager:

This question has been retained from the PISA 2009 Main Survey (StQ13b). This item, in combination with StQ13a, establishes the father of the student’s main occupation.

Use examples that match the job title examples given in Q13a.

Notes for Translator:

This question was worded identically in the PISA 2009 Main Survey (StQ13b).

Q13: Notes for Test Administrator:

‘main job’: If the father has more than one job, his ‘main job’ is the job in which most time is spent, not necessarily the highest earning job.

If the father is working at home, doing home duties, please instruct the student to describe this as ‘working at home, doing home duties’.

If a student’s father (or equivalent male guardian) is deceased, then his last job can still be provided - but if this is unknown the question should be left blank.

Encourage students to answer this question. A general description such as ‘works in an office’ is better than nothing written.

Q14	What is the <highest level of schooling> completed by your father?
	If you are not sure how to answer this question, please ask the <test administrator> for help. (Please tick only one box.)
a)	<ISCED level 3A>	1
b)	<ISCED level 3B, 3C>	2
c)	<ISCED level 2>	3
d)	<ISCED level 1>	4
e)	He did not complete <ISCED level 1>	5

Notes for National Project Manager:

This question has been retained from the PISA 2009 Main Survey (StQ14). This item establishes the father of the student’s highest level of education.

Stem: <highest level of schooling> should be adapted to refer to the sections of schooling that correspond to ISCED levels 1 to 3.

The categories need to be specified using country-specific terms that will be understood by the students responding to the question. Each category needs to be mapped to the ISCED classification of educational levels (see Classifying Educational Programmes — Manual for ISCED-97 Implementation in OECD Countries, 1999 Edition, OECD). The following information about ISCED is taken from this manual.

ISCED Level 1: Primary level of education.

ISCED Level 2: Lower secondary level of education.

ISCED Level 3A: Upper secondary level of education with programmes designed to provide direct access to ISCED 5A.

ISCED Level 3B: Upper secondary level of education with programmes designed to provide direct access to ISCED 5B. Level 3B tends to be more practical and has a vocational orientation.

ISCED Level 3C: Upper secondary level of education with programmes designed to provide direct access to the labour market.

In some countries there may not be an administrative or structural boundary between ISCED 2 and 3 in the educational system. In these cases one may ask about completion of the grade/year level that can be defined as an implicit boundary between ISCED level 2 and 3.

Adaptations to this question are a requirement. They need to be described in the Questionnaire Adaptation Spreadsheet and agreed between the NPM and the Consortium.

Notes for Translator:

This question was worded identically in the PISA 2009 Main Survey (StQ14).

Do NOT use the expressions ‘ISCED 3A’, ‘ISCED 3B’ etc. as the respondent will not understand them.

Q14: Notes for Test Administrator:

This question is asking about completion, that is, obtaining a qualification. Just attending an institution where these qualifications can be obtained is not enough.

If a student’s father (or equivalent male guardian) is deceased, then his qualifications can still be provided – but if this is unknown the question should be left blank.

If the student’s father obtained his qualifications abroad, please help the student to choose the response from the list that is closest. Asking questions like “How long did he go to school for?” and “Did he go to university?” should help clarify.

Q15	Does your father have any of the following qualifications?
	If you are not sure which box to choose, please ask the <test administrator> for help. (Please tick one box in each row.)
		Yes	No
a)	<ISCED level 6>	1	2
b)	<ISCED level 5A>	1	2
c)	<ISCED level 5B>	1	2
d)	<ISCED level 4>	1	2

Notes for National Project Manager:

This question has been retained from the PISA 2009 Main Survey (StQ15). This item establishes whether or not the father of the student has certain educational qualifications.

The categories need to be specified using country-specific terms that will be understood by the students. Each qualification needs to be mapped to the ISCED classification of educational levels (see Classifying Educational Programmes — Manual for ISCED-97 Implementation in OECD Countries, 1999 Edition, OECD). The following information about ISCED is taken from this manual.

ISCED Level 6: Advanced research qualification, devoted to advanced study and original research, requiring submission of a thesis or dissertation of publishable quality.

ISCED Level 5A: Qualification obtained from a tertiary study programme with a strong theoretical foundation typically with a minimum duration of three years’ full time equivalent, providing entry into a profession with high skills requirements or an advanced research programme.

ISCED Level 5B: Qualification obtained in tertiary programmes that are generally more practical/technical/occupationally specific and typically shorter than ISCED 5A programmes. Typically, these programmes have a minimum of two years’ full-time equivalent duration and prepare students to enter a particular occupation.

ISCED Level 4: Qualification obtained in programmes that overlap the boundary between upper-secondary and post-secondary education. They are typically not significantly more advanced than programmes at Level 3 and have a full-time equivalent duration of between 6 months and 2 years.

Adaptations to this question are a requirement. They need to be described in the Questionnaire Adaptation Spreadsheet and agreed between the NPM and the Consortium.

Notes for Translator:

This question was worded identically in the PISA 2009 Main Survey (StQ15).Do NOT use the expressions ‘ISCED 5A’, ‘ISCED 5B’ etc. as the respondent will not understand them.

Q15: Notes for Test Administrator

This question is asking about completion; that is, obtaining a qualification. Just attending an institution where these qualifications can be obtained is not enough.

If a student’s father (or equivalent male guardian) is deceased, then his qualifications can still be provided – but if this is unknown the question should be left blank.

If a student has a father (or equivalent male guardian) who was trained in a non-university setting but currently has an occupation requiring university education for admission (common examples are teaching, nursing, and some accounting occupations) he or she may use response category b).

If the student’s father obtained his qualifications abroad, please help the student to choose the response from the list that is closest. Asking questions like “Did he go to university?” should help clarify.

Q16	What is your father currently doing?
	(Please tick only one box.)
a)	Working full-time <for pay>	1
b)	Working part-time <for pay>	2
c)	Not working, but looking for a job	3
d)	Other (e.g. home duties, retired)	4

Notes for National Project Manager:

This question has been retained from the PISA 2009 Main Survey (StQ16).

This item provides information on the father of the student’s current employment status; as well as additional information on social background.

Adaptations to this question are a requirement. They need to be described in the Questionnaire Adaptation Spreadsheet and agreed between the NPM and the Consortium.

Notes for Translator:

This question was worded identically in the PISA 2009 Main Survey (StQ16).

Q16: Notes for Test Administrator:

Students should answer this question with regard to their father’s main occupation. If e.g. the father is working part-time but also doing home duties, students should tick b) ‘working part-time’.

If a student has no father (or equivalent male guardian), then he or she should leave this question blank.

Please note that students should tick only one box.