Pilot test results summary

Introduction

This report describes the performance of the items previously pilot tested that are being suggested for operational use in the student core questionnaire for 2013. Note that in the 2012 pilot, the grades 8 and 12 student core items were given as part of two assessments (reading and mathematics). Therefore, data are presented separately from the two pilot administrations.

Income/Wealth (Pilot Items): Other Household Resources

In order to achieve a more discriminating socioeconomic status (SES) measure, the NAEP SES expert panel made recommendations that led to development of a new item of household resources that may be associated with SES. These additional items focus on the size of the student’s home and the family’s possession of labor-saving devices that are more common among higher income than lower income families. Question 24 on the 2009 Program for International Student Assessment (PISA) student questionnaire asked students to report on home possessions (summing over 17 items including three different book types: poetry, school work aids, and reference/manuals). There is a long history of evidence of academic achievement correlated with SES when using home possessions as a measure of income/wealth in an SES composite (Cuff 1934).

Grade 4, 8, 12 - Pilot

VE592166

Do you have the following items at your home? Fill in ovals for all that apply.

A Access to the Internet

B Clothes dryer just for your family

C Dishwasher

D More than one bathroom

E Your own bedroom

Grade 4

	Item Statistics
Response Options	G4 Writing Pilot 2012
	Pct
Access to the Internet	76
No response	24
Clothes dryer just for your family	76
No response	24
Dishwasher	71
No response	29
More than one bathroom	69
No response	31
Your own bedroom	68
No response	32
Non-responses	0

Grade 8

	Item Statistics	Item Statistics
Response Options	G8 Mathematics Pilot 2012	G8 Reading Pilot 2012
	Pct	Pct
Access to the Internet	88	87
No response	12	13
Clothes dryer just for your family	83	84
No response	17	16
Dishwasher	70	68
No response	30	32
More than one bathroom	73	73
No response	27	27
Your own bedroom	72	71
No response	28	29
Non-responses	0	0

Grade 12

	Item Statistics	Item Statistics
Response Options	G12 Mathematics Pilot 2012	G12 Reading Pilot 2012
	Pct	Pct
Access to the Internet	95	93
No response	5	7
Clothes dryer just for your family	92	90
No response	8	10
Dishwasher	78	76
No response	22	24
More than one bathroom	82	82
No response	18	18
Your own bedroom	82	80
No response	18	20
Non-responses	0	0

Analysis

• Quality and Performance: data on missing rates and response patterns did not reveal any item quality problems.

  • Pilot response patterns and missing rates for the other household resources item did not suggest any within-grade item quality problems. However, across grade analysis reveals unexpected differences. For example: Access to the Internet (Math: 4/8/12): 76% / 87% / 95% and Clothes dryer just for your family (Math: 4/8/12): 76% / 84% / 92%. That said, there are several competing explanations that are unexplainable with the data we have:

    • Older children may be part of families that have more children, which might warrant greater access to the Internet (for educational and entertainment purposes) as well as greater need for a clothes dryer.

    • Older children may be part of families that have older parents and older parents may have greater income and wealth, which would allow for disposable income to purchase additional home possessions such as Internet or a clothes dryer.

    • Younger students may be unaware of access to certain home possessions and because this is a “select all that apply” item, there is no way to distinguish those who didn’t select the response option because “I don’t know” is not an option here or because the answer is “No.”

  • Rationale for inclusion: Household resources item under consideration by NAEP SES expert panel for potential SES composite variable and/or proxy for family income/wealth.

  • Post-analysis revision:

    • Remove the word “items” from the question stem so that it reads “Do you have the following in your home?” [Previous version created challenge with Spanish translation. Reviewers and translators determined the inclusion of the word “item” in the sentence was unnecessary.]

Income/Wealth (Pilot Items): Household Composition

The NAEP SES expert panel suggested including measures of household composition in NAEP as a measure of SES based partly on the dilution model which posits that, given the finite parental resources, as the number of children in the family increases, the resources accrued by any one child necessarily decline (Downey 2001). A negative relationship between family size and children’s academic-related achievement has been demonstrated in many studies (Blake 1981; Greenberg 1985; Hanushek 1992; Hauser and Sewell 1986; Hill and O' Neill 1997; Sputa and Paulson 1995). In general, these studies found that children from larger families have lower academic performance than children from smaller families.

The SES panel also noted that family structure or whether there are one or two parents in the household will have an opposite effect in that two parents provide more supporting resources than one. Compared to children living in homes with both biological parents, peers living in single-mother or step-father families were more likely to have ever been suspended or expelled from school; more likely to have engaged in delinquent activities; more likely to have problems getting along with their teachers, doing homework, and paying attention in school; and more likely to have lower grade point averages (Manning & Lamb 2003). Children from single-parent or stepparent families often report that their parents had lower educational expectations for them, compared to reports from children in families with both biological parents. These parents are less likely to monitor their children’s schoolwork and provide less overall supervision on children’s social activities, compared to reports from children in families with both biological parents (Astone & McLanahan 1991). On average, compared to peers raised in blended or single-parent families, students from families with both biological parents completed more years of schooling and were also more likely to graduate from high school and, attend college (Ginther & Pollak 2004).

Grade 4, 8, 12 - Pilot

VE592165

Do the following people live in your home? Fill in ovals for all that apply.

A Mother

B Stepmother

C Foster mother

D Other female legal guardian

E Father

F Stepfather

G Foster father

H Other male legal guardian

Grade 4

	Item Statistics
Response Options	G4 Writing Pilot 2012
	Pct
Mother	91
No response	9
Stepmother	4
No response	96
Foster mother	1
No response	99
Other female legal guardian	11
No response	89
Father	69
No response	31
Stepfather	10
No response	90
Foster father	1
No response	99
Other male legal guardian	10
No response	90
Non-responses	0

Grade 8

	Item Statistics	Item Statistics
Response Options	G8 Mathematics Pilot 2012	G8 Reading Pilot 2012
	Pct	Pct
Mother	89	88
No response	11	12
Stepmother	6	4
No response	94	96
Foster mother	1	1
No response	99	99
Other female legal guardian	11	12
No response	89	88
Father	62	59
No response	38	41
Stepfather	14	13
No response	86	87
Foster father	1	1
No response	99	99
Other male legal guardian	8	9
No response	92	91
Non-responses	0	0

Grade 12

	Item Statistics	Item Statistics
Response Options	G12 Mathematics Pilot 2012	G12 Reading Pilot 2012
	Pct	Pct
Mother	90	88
No response	10	12
Stepmother	4	4
No response	96	96
Foster mother	0	0
No response	100	100
Other female legal guardian	5	5
No response	95	95
Father	63	64
No response	37	36
Stepfather	13	12
No response	87	88
Foster father	0	0
No response	100	100
Other male legal guardian	3	4
No response	97	96
Non-responses	0	0

Analysis

• Quality and Performance:

  • One percent (or less) students reported having foster parents. On NDE, when reported by states, there will be some cases where less than 62 respondents will be in the cell and in those cases NDE will place a flag noting that estimates cannot be reported for students in that category.

• Rationale for inclusion: Item under consideration by SES panel for potential SES composite variable.

• Post-analysis revision:

  • Merge response choices: Foster mother and other female legal guardian

  • Merge response choices: Foster father and other male legal guardian

Post High School Preparation

Research has shown that educational aspirations are associated with high-school academic performance (Dumais 2002; Goldsmith 2004; Lee and Bryk 1988; Lynch 2000). Nonetheless, the Governing Board was dissatisfied with the item quality of past NAEP measures of students’ plans after college. Research has often shown that the reason for these misaligned ambitions is preparation (Schneider and Stevenson 1999). While students are very ambitious and have high aspirations, the better predictor of their academic achievement is receiving information, strategies, and mentoring (from parents and others) to help them think about and prepare for careers and postsecondary education (Schneider and Stevenson 1999, McDonough 1997; Ream and Rumberger 2008; Sokatch 2006; Wimberly & Noeth 2004). The following pilot item was developed based on the aforementioned cited research showing a connection between mentoring, aligned ambitions, post-high school preparation and student achievement.

Grade 12 - Pilot

VE627773

During this school year, which of the following have you done? Fill in ovals for all that apply.

A Taken the SAT or ACT College Entrance Exams

B Submitted the Free Application for Federal Student Aid (FAFSA)

C Applied to a 2-year college

D Been accepted to a 2-year college

E Applied to a 4-year college

F Been accepted to a 4-year college

G Applied to a technical training program (such as electrician,
beautician, mechanic, computer programmer, etc.)

H Been accepted to a technical training program

I Talked with a military recruiter

J Been accepted into the military

K Applied for a full-time job

L Been interviewed for a full-time job

M None of the above

Grade 12

	Item Statistics	Item Statistics
Response Options	G12 Mathematics Pilot 2012	G12 Reading Pilot 2012
	Pct	Pct
Taken the SAT or ACT College Entrance Exams	61	66
No response	39	34
Submitted the Free Application for Federal Student Aid (FAFSA)	33	34
No response	67	66
Applied to a 2-year college	20	18
No response	80	82
Been accepted to a 2-year college	8	9
No response	92	91
Applied to a 4-year college	56	60
No response	44	40
Been accepted to a 4-year college	40	45
No response	60	55
Applied to a technical training program (such as electrician, beautician, mechanic, computer programmer, etc.)	4	4
No response	96	96
Been accepted to a technical training program	2	3
No response	98	97
Talked with a military recruiter	16	15
No response	84	85
Been accepted into the military	2	2
No response	98	98
Applied for a full-time job	9	8
No response	91	92
Been interviewed for a full-time job	5	5
No response	95	95
None of the above	12	8
No response	88	92
Non-responses	0	0

Analysis

• Quality and Performance: Pilot response patterns and missing rates for this item did not suggest any within-grade or across-grade item quality problems.

• Rationale for inclusion:

  • Item replaces previous G12 items related to post HS aspirations, which were requested for removal by NAGB in 2010. New item measures behavior, which is expected to be a more valid predictor of average scale scores and reflection of post high school preparation.

NAEP Pilot Information for New Operational Items – 2013 Wave 1 Page 9