Download:
pdf |
pdfNational Longitudinal Survey of
Youth 1997 (NLSY97)
Technical Sampling Report
Whitney Moore, Steven Pedlow,
Parvati Krishnamurty, and Kirk Wolter
National Opinion Research Center
Chicago, IL
November 2000
TABLE OF CONTENTS
Page
ACKNOWLEDGMENTS ......... ............ ............ ............ ............ ............ ............ ............ ......... 1
LIST OF TABLES AND FIGURES....... ............ ............ ............ ............ ............ ............ .......... 2
CHAPTER 1: INTRODUCTION ........... ............ ............ ............ ............ ............ ............ ........ 11
CHAPTER 2: DESIGN OF THE NLSY97 SAMPLE...... ............ ............ ............ ............ ........ 12
2.1
The Eligible Population ............ ............ ............ ............ ............ ............ ............ ........ 12
2.2
Overview of the Sample ............ ............ ............ ............ ............ ............ ............ ........ 14
2.3
Screening Questions and Forms. ............ ............ ............ ............ ............ ............ ........ 18
CHAPTER 3: SELECTION OF THE NLS97 SAMPLE . ............ ............ ............ ............ ........ 20
3.1
Selection of the Cross-Sectional (CX) Sample .... ............ ............ ............ ............ ........ 20
3.2
Selection of the Supplemental (SU) Sample........ ............ ............ ............ ............ ........ 24
CHAPTER 4: WEIGHTING THE SAMPLE ...... ............ ............ ............ ............ ............ ........ 29
CHAPTER 5: ACCURACY AND PRECISION OF THE SAMPLE RESULTS ... ............ ........ 38
5.1
Potential Impact of Nonresponse ............ ............ ............ ............ ............ ............ ........ 38
5.2
Refielding......... ............ ............ ............ ............ ............ ............ ............ ............ ........ 46
5.3
Representativeness of the Sample........... ............ ............ ............ ............ ............ ........ 49
5.4
Estimates of the NLSY97 Shortfall ........ ............ ............ ............ ............ ............ ........ 71
5.5
Estimates of Variance ... ............ ............ ............ ............ ............ ............ ............ ........ 81
CHAPTER 6: SUMMARY OF THE ACCURACY AND REPRESENTATIVENESS
OF THE NLSY97 SAMPLES ... ............ ............ ............ ............ ............ ............ ............ ........ 96
APPENDIX A: Construction of Demographic Variables for the NLSY97 ............ ............ ........ 99
APPENDIX B: This appendix is blank... ............ ............ ............ ............ ............ ............ ...... 111
APPENDIX C: This appendix is blank... ............ ............ ............ ............ ............ ............ ...... 112
APPENDIX D: Analysis of Language Barrier Cases ...... ............ ............ ............ ............ ...... 113
APPENDIX E: Review of Proxy Interviews ...... ............ ............ ............ ............ ............ ...... 131
2
APPENDIX F: PSU-Level Socio-Economic-Status Analysis ....... ............ ............ ............ ...... 135
APPENDIX G: Comparison of the NLSY97 Family Income Distribution to the CPS
Family Income Distribution ....... ............ ............ ............ ............ ............ ...... 141
APPENDIX H: Introduction and Paper Screener ............ ............ ............ ............ ............ ...... 156
APPENDIX I: Introduction and Age Questions from Extended Screener.. ............ ............ ...... 160
APPENDIX J: Proxy Introduction and Paper Screener .... ............ ............ ............ ............ ...... 167
APPENDIX K: Advance Letters and Brochure ... ............ ............ ............ ............ ............ ...... 170
APPENDIX L: Review of Gatekeeper Interviews............ ............ ............ ............ ............ ...... 177
APPENDIX M: NIR Reports NLSY97 Round 1. ............ ............ ............ ............ ............ ...... 181
APPENDIX N: Report on NLSY97 Computer Crashes ... ............ ............ ............ ............ ...... 196
APPENDIX O: Report on NLSY97 Debriefing Interviews .......... ............ ............ ............ ...... 204
APPENDIX P: Analysis of Interview Validation ............ ............ ............ ............ ............ ...... 215
APPENDIX Q: Analysis of Parent Education for the NLSY97 .... ............ ............ ............ ...... 220
APPENDIX R: Two Comparisons of Expected vs. Observed Youth Counts ......... ............ ...... 226
APPENDIX S: Adjusted Screener Weights Used for Age Distribution Analysis ... ............ ...... 243
APPENDIX T: Urban/Rural Age Distributions... ............ ............ ............ ............ ............ ...... 247
3
Acknowledgments
We would like to acknowledge the help of the following people in preparing this report: Michael
Horrigan, Bureau of Labor Statistics; Clessen J. Martin and Linda Curran, Defense Manpower Data
Center; and Gail Hoff and Harrison Greene, National Opinion Research Center. Some of the appendices
have been written by Rashna Ghadialy, Lisa Lee, and Sally Murphy of the National Opinion Research
Center.
1
List of Tables and Figures
Tables
Page
2.1 Target Sample Sizes for the NLSY97........... ............ ............ ............ ............ ............ ........ 14
2.2 Sizes of the CX and SU Samples ..... ............ ............ ............ ............ ............ ............ ........ 15
2.3 Results of Screening Operations ...... ............ ............ ............ ............ ............ ............ ........ 16
2.4 Achieved Sample Sizes for the NLSY97 ...... ............ ............ ............ ............ ............ ........ 17
3.1 Distribution of CX PSUs, by Census Region and Metropolitan Status ............ ........ 20
3.2 Projected Screening Sample Sizes and Yields Used in Planning the NLSY97 ............ ........ 23
3.3 Actual Screening Sample Sizes........ ............ ............ ............ ............ ............ ............ ........ 24
3.4 Sample Sizes and Sampling Rates Used in Determining the
Measure for Size of the SU Sample . ............ ............ ............ ............ ............ ............ ........ 25
3.5 Distribution of SU PSUs, by Census Region and Metropolitan Status ............ ............ ........ 25
3.6 Summary of the Design of the CX and SU Samples . ............ ............ ............ ............ ........ 28
4.1 Unadjusted and Capped Missed Housing Unit Adjustments .. ............ ............ ............ ........ 31
4.2 8s Used for Precision Weighting to Combine the CX and SU Samples ........... ............ ........ 33
4.3 Cell (Weighted) Response Rates (in Percent) Used for the NLSY97
Nonresponse Weighting Adjustment ............ ............ ............ ............ ............ ............ ........ 34
4.4 Population Totals Used for NLSY97 Poststratification ......... ............ ............ ............ ........ 35
4.5 Coverage Ratios for NLSY97 Sample .......... ............ ............ ............ ............ ............ ........ 36
4.6 Summary Table of Housing Unit and Household Weights..... ............ ............ ............ ........ 37
4.7 Summary Table for Youth Weights . ............ ............ ............ ............ ............ ............ ........ 38
5.1 Final Response Status for All Screeners ....... ............ ............ ............ ............ ............ ........ 38
5.2 Screener Completion Rates. ............ ............ ............ ............ ............ ............ ............ ........ 39
5.3 Final Response Status for NLSY97 Youths.. ............ ............ ............ ............ ............ ........ 39
5.4 ASVAB Test Status for NLSY97 Youths..... ............ ............ ............ ............ ............ ........ 40
5.5 Summary of Response Rates (in Percent), Overall and by Domain .... ............ ............ ........ 41
2
5.6 Distributions (Unweighted) of Screener Variables for CPS Youths, Completed
Screener Respondents, and NLSY97 Interview and Test Completes .. ............ ............ ........ 45
5.7 True Overall Proportions of Youths Given Varying Proportions for Nonparticipants and
Varying Response Rates ..... ............ ............ ............ ............ ............ ............ ............ ........ 46
5.8 Screener Response Status Before and After Refielding.......... ............ ............ ............ ........ 47
5.9 Screener Response Status Before and After Refielding: CX Sample . ............ ............ ........ 48
5.10 Screener Response Status Before and After Refielding: SU Sample ............ ............ ........ 48
5.11 Response Status for the Youth Interview Before and After Refielding.......... ............ ........ 48
5.12 Weighted-up 1990 Census Counts of Youths Under Age 18 and of Housing
Units for the CX Segments ............ ............ ............ ............ ............ ............ ........ 50
5.13 Weighted-up 1990 Census Counts of Minority Youths Under Age 18
for the SU Segments .......... ............ ............ ............ ............ ............ ............ ............ ........ 51
5.14 Weighted-up Listing Counts for the CX Sample ..... ............ ............ ............ ............ ........ 52
5.15 Comparison of Census and Listing Counts in the SU Sample........... ............ ............ ........ 54
5.16 Sums of Base Weights for Selected Housing Units . ............ ............ ............ ............ ........ 54
5.17 Comparison of the NLSY97 Vacancy Status of Housing Units ........ ............ ............ ........ 55
5.18 Effect of the Weighting Caps on the Sums of the Household Weights .......... ............ ........ 56
5.19 Distribution of Households by Reported Number of Youths Aged 12 to 16.. ............ ........ 58
5.20 Conditional Distribution of Households (HHs) by Reported Number of Youths Aged
12 to 16, Given at Least One Such Youth... ............ ............ ............ ............ ............ ........ 59
5.21 Conditional Mean Youths, Age 12 to 16, Per Household..... ............ ............ ............ ........ 60
5.22 Average Household Size .. ............ ............ ............ ............ ............ ............ ............ ........ 60
5.23 Distribution of Households by Size of Household... ............ ............ ............ ............ ........ 61
5.24 Youths as a Proportion of Total Persons Screened .. ............ ............ ............ ............ ........ 61
5.25 Race by Gender Percentages for CPS vs. NLSY97-Eligible Sample ............ ............ ........ 62
5.26 Race by Gender Percentages for CPS vs. NLSY97 Interviews ......... ............ ............ ........ 62
5.27 Race by Gender Percentages for CPS vs. NLSY97 Test Completes . ............ ............ ........ 63
3
5.28 Estimated 1990 Census Coverage Ratios (March, 1998)...... ............ ............ ............ ........ 66
5.29 Estimated CPS Coverage Ratios (March, 1998) ...... ............ ............ ............ ............ ........ 67
5.30 Estimated NLSY97/PAY97 Coverage Ratios.......... ............ ............ ............ ............ ........ 68
5.31 Analysis of Actual versus Expected Youths in CX Screening Sample: Method 1A ... ........ 73
5.32 Analysis of Actual versus Expected Youths in SU Screening Sample: Method 1A.. ........ 75
5.33 Analysis of Actual versus Expected Youths in NLSY97 Screening Sample
(CX + SU): Method 1A.... ............ ............ ............ ............ ............ ............ ............ ........ 76
5.34 Estimates of the Shortfall of Youths Aged 12-16 in the NLSY97 Screening
Sample: Method 2 ........... ............ ............ ............ ............ ............ ............ ............ ........ 77
5.35 Expected Youths versus Actual Youths Screened: Method 3A......... ............ ............ ........ 78
5.36 Five Estimates of the Shortfall of Eligible Youths .. ............ ............ ............ ............ ........ 78
5.37 Variance Estimates for Overall NLSY97 Sample: Proportions ....... ............ ............ ........ 83
5.38 Variance Estimates for Overall NLSY97 Sample: Means ... ............ ............ ............ ........ 84
5.39 Variance Estimates for NLSY97 Males: Proportions ........... ............ ............ ............ ........ 85
5.40 Variance Estimates for NLSY97 Males: Means ...... ............ ............ ............ ............ ........ 86
5.41 Variance Estimates for NLSY97 Females: Proportions........ ............ ............ ............ ........ 87
5.42 Variance Estimates for NLSY97 Females: Means... ............ ............ ............ ............ ........ 88
5.43 Variance Estimates for NLSY97 Hispanics: Proportions ..... ............ ............ ............ ........ 89
5.44 Variance Estimates for NLSY97 Hispanics: Means ............ ............ ............ ............ ........ 90
5.45 Variance Estimates for NLSY97 Non-Hispanic Blacks: Proportions ............ ............ ........ 91
5.46 Variance Estimates for NLSY97 Non-Hispanic Blacks: Means........ ............ ............ ........ 92
5.47 Variance Estimates for NLSY97 Non-Hispanic Nonblacks: Proportions ...... ............ ........ 93
5.48 Variance Estimates for NLSY97 Non-Hispanic Nonblacks: Means ............ ............ ........ 94
5.49 Comparison of Summary Statistics of NLSY97 DEFTS ..... ............ ............ ............ ........ 95
A-1
Crosstabulation of Screener Hispanic Origin by OLQ Ethnicity (recoded) ............ ...... 101
A-2
Crosstabulation of Screener Hispanic Origin by OLQ Origin (recoded).... ............ ...... 102
A-3
Final Hispanic Origin Frequencies ......... ............ ............ ............ ............ ............ ...... 102
4
A-4
Crosstabulation of Screener Sex by Youth Questionnaire Sex...... ............ ............ ...... 104
A-5
Crosstabulation of Screener Sex by Parent Questionnaire Sex...... ............ ............ ...... 104
A-6
Final Sex Frequencies ... ............ ............ ............ ............ ............ ............ ............ ...... 104
A-7
Crosstabulation of Screener Race by OLQ Origin (recoded) ........ ............ ............ ...... 106
A-8
Final Race Frequencies . ............ ............ ............ ............ ............ ............ ............ ...... 106
A-9
Crosstabulation of OLQ Year of Birth by Screener Year of Birth ............ ............ ...... 109
A-10
Crosstabulation of Youth Questionnaire Year of Birth by Screener Year of Birth . ...... 109
A-11
Age Frequencies............ ............ ............ ............ ............ ............ ............ ............ ...... 110
D-1
Distribution (Percent) of American Youths by Ability to Speak English
and by Hispanic Origin . ............ ............ ............ ............ ............ ............ ............ ...... 115
D-2
Summary of Total Student and LEP Enrollments, 1996-97 .......... ............ ............ ...... 116
D-3
Number and Percent of LEP Students in Each Grade Level.......... ............ ............ ...... 117
D-4
Percentage of LEP Students by Language Group, 1991-92........... ............ ............ ...... 118
D-5
Percentages of LEP Students Identified and Excluded from the NAEP Sample ..... ...... 119
D-6
ETP Status* Sample Crosstabulation....... ............ ............ ............ ............ ............ ...... 121
D-7
ETP: OOS, Language Barrier* Hispanic Origin Crosstabulation... ............ ............ ...... 122
D-8
STP Status *
D-9
STP: OOS, Language Barrier* Hispanic Origin Crosstabulation ... ............ ............ ...... 124
D-10
Youths Screened into the ETP: PSUs with Proportion Language Barrier
Sample Crosstabulation ............ ............ ............ ............ ............ ...... 123
Greater Than or Equal to 0.10.... ............ ............ ............ ............ ............ ............ ...... 128
D-11
Estimates of English Language Deficiency ......... ............ ............ ............ ............ ...... 129
E-1
Review of Proxy Record of Calls ........... ............ ............ ............ ............ ............ ...... 134
F-1
PSU-level Socio-Economic Correlations: Education ...... ............ ............ ............ ...... 136
F-2
PSU-level Socio-Economic Correlations: Income........... ............ ............ ............ ...... 138
G-1
Total NLSY97 Household Wages and Salary Income...... ............ ............ ............ ...... 148
G-2
Total NLSY97 Household Self-Employment Income ...... ............ ............ ............ ...... 148
G-3
Total NLSY97 Household Earned Income (Wages/Salary + Self-Employment).... ...... 148
5
G-4
Total NLSY97 Household Income (All Sources) ............ ............ ............ ............ ...... 148
G-5
Total NLSY97 Household Wages and Salary Income: Cross-Sectional Sample ... ...... 149
G-6
Total NLSY97 Household Self-Employment Income: Cross-Sectional Sample.... ...... 149
G-7
Total NLSY97 Household Earned Income: Cross-Sectional Sample........ ............ ...... 149
G-8
Total NLSY97 Household Income: Cross-Sectional Sample ....... ............ ............ ...... 149
G-9
Total CPS Household Wages and Salary Income ............ ............ ............ ............ ...... 150
G-10
Total CPS Household Self-Employment Income ............ ............ ............ ............ ...... 150
G-11
Total CPS Household Earned Income ... ............ ............ ............ ............ ............ ...... 150
G-12
Total CPS Household Income.... ............ ............ ............ ............ ............ ............ ...... 150
G-13
Total CPS Household Wages and Salary Income (Reported Data Only) ... ............ ...... 151
G-14
Total CPS Household Self-Employment Income (Reported Data Only).... ............ ...... 151
G-15
Total CPS Household Earned Income (Reported Data Only)........ ............ ............ ...... 151
G-16
Total CPS Household Income (Reported Data Only) ....... ............ ............ ............ ...... 151
G-17
Total NLSY97 Household Wages and Salary Income by Race/Ethnicity .. ............ ...... 152
G-18
Total NLSY97 Household Self-Employment Income by Race/Ethnicity .. ............ ...... 152
G-19
Total NLSY97 Household Earned Income by Race/Ethnicity ...... ............ ............ ...... 153
G-20
Total NLSY97 Household Income by Race/Ethnicity...... ............ ............ ............ ...... 153
G-21
Total CPS Household Wages and Salary Income by Race/Ethnicity ......... ............ ...... 154
G-22
Total CPS Household Self-Employment Income by Race/Ethnicity.......... ............ ...... 154
G-23
Total CPS Household Earned Income by Race/Ethnicity. ............ ............ ............ ...... 155
G-24
Total CPS Household Income by Race/Ethnicity ............ ............ ............ ............ ...... 155
L-1
Review of Gatekeeper Record of Calls... ............ ............ ............ ............ ............ ...... 180
M-1
NIR Sample Questions Divided Into Three Main Sections ........... ............ ............ ...... 185
M-2
Reasons for a Non-Interview .... ............ ............ ............ ............ ............ ............ ...... 187
M-3
Comparison NIR Reports Due to Refusals .......... ............ ............ ............ ............ ...... 188
M-4
All Inhabited Households Compared to Those Where Potentially
Eligible Youth Were Identified.. ............ ............ ............ ............ ............ ............ ...... 190
6
M-5
Composition of NIR Household ............ ............ ............ ............ ............ ............ ...... 191
M-6
Ratio of Youths to Household Units and to Household Members. ............ ............ ...... 192
M-7
Determination of the Number of Potentially Eligible Youths ....... ............ ............ ...... 192
N-1
PSU 721 ..... .................... .................... ................... .................... .................... ................... .................... .................... ............. 198
N-2
PSU 764 ..... .................... .................... ................... .................... .................... ................... .................... .................... ............. 199
N-3
PSU 795 ..... .................... .................... ................... .................... .................... ................... .................... .................... ............. 200
N-4
PSU 862 ..... .................... .................... ................... .................... .................... ................... .................... .................... ............. 201
N-5
Location of Largest Numbers of Crashed/Technical Problems and “No HU” Cases .... 203
O-1
Number of Times a Computer Problem Prevented Completion
of a Screening Interview. .................... ................... .................... .................... ................... .................... ..............207
O-2
Types of Computer Problems Experienced by FIs . .................... .................... ................... .............. 207 .................... ................... ....................
O-3
Computer Problems in NLSY97 and Prior CAPI Experience ..... .................... ................... .............. 208
O-4
What FIs Told Informants About Who is Eligible to Participate in NLSY97 . ............. 208
O-5
Number of FIs Giving Each Rating of Difficulty in Gaining Cooperation ...... ................... .............. 208 ....................
O-6
FIs’ Ratings on How Often Informants Read Advance Letter and Brochure .. ............. 209....................
O-7
FIs’ Ratings on How Often Informants Did Not Report Accurately
on Household Roster ...... .................... ................... .................... .................... ................... .................... .................... ............. 209
O-8
FI Reports of Questions and Concerns Expressed by NLSY97 Informants..... ............. 209
P-1
NLSY97 Round 1 Validation Interviews ............ ............ ............ ............ ............ ...... 218
P-2
NLSY97 Round 1 Validation Interviews Ineligible Screeners Only .......... ............ ...... 218
P-3
NLSY97 Round 1 Validation Interview Respondent Comments Regarding the
Survey or Experience with Interviewer .. ............ ............ ............ ............ ............ ...... 219
P-4
Elapsed Time Between Screener and Validation Interview for 105
Screeners for Which Validation was Other Than Successful ........ ............ ............ ...... 219
Q-1
CPS and NLSY97 Components of Final Grade Categories........... ............ ............ ...... 222
Q-2
Youths by Presence of Parent – Comparison of Data Sets ............ ............ ...... 223
Q-3
Extent of Missingness of the Data .......... ............ ............ ............ ............ ............ ...... 223
7
Q-4
Father’s Highest Grade Attained ............ ............ ............ ............ ............ ............ ...... 224
Q-5
Mother’s Highest Grade Attained ........... ............ ............ ............ ............ ............ ...... 224
R-1
Expected Counts of Youths Aged 12-16 to be Screened .. ............ ............ ............ ...... 229
R-2
Actual Counts of Youths Aged 12-16 Screened .. ............ ............ ............ ............ ...... 229
R-3
Youths Aged 12-16 Actually Screened as a Percent of Expected Youths.. ............ ...... 229
R-4
Some PSU-Level Percentile Statistics on Percentage Variables
(Percentage of Expected Youths Screened) ......... ............ ............ ............ ............ ...... 232
R-5
Classification of Four PSU-Level Percentage Variables .. ............ ............ ............ ...... 234
R-6
Expected Youths Aged 12-16 to be Screened for 10 “Lowest” CX and SU PSUs.. ...... 235
R-7
Actual Youths Aged 12-16 Screened for 10 “Lowest” CX and SU PSUs.. ............ ...... 236
R-8
Actual Youths Aged 12-16 Screened as a Percentage of Expected
for 10 “Lowest” CX and SU PSUs ......... ............ ............ ............ ............ ............ ...... 237
S-1
The Four Groups Used in Step 2b of the Age-Distribution Weighting ...... ............ ...... 245
S-2
λs Used for Precision Weighting for Age-Distribution Weights ... ............ ............ ...... 246
T-1
Unweighted NLSY97 Age Distributions for Urban/Rural Groups ............ ............ ...... 253
T-2
Weighted NLSY97 Age Distributions for Urban/Rural Groups.... ............ ............ ...... 254
8
Figures
Page
2.1 Map of CX and SU Samples for NLSY97 ..... ............ ............ ............ ............ ............ ........ 17
5.1 Screener Completion by PSU ........... ............ ............ ............ ............ ............ ............ ........ 42
5.2 Distribution of Screener Type A1 Nonresponse by PSU......... ............ ............ ............ ........ 42
5.3 Screener Completion by Segment..... ............ ............ ............ ............ ............ ............ ........ 42
5.4 Distribution of Screener Type A1 Nonresponse by Segment .. ............ ............ ............ ........ 42
5.5 Weighted Counts of Persons, Total Sample... ............ ............ ............ ............ ............ ........ 57
5.6 Weighted Counts of Persons, CX Sample ..... ............ ............ ............ ............ ............ ........ 58
5.7 Weighted Counts of Persons, SU Sample...... ............ ............ ............ ............ ............ ........ 58
D-1 ETP: Language Barrier versus Hispanic Origin, PSU Level . ............ ............ ............ ...... 125
D-2 ETP: Language Barrier versus Hispanic Origin, PSU Level, Cross-Sectional Sample ...... 126
D-3 ETP: Language Barrier versus Hispanic Origin, PSU Level, Supplemental Sample ... ...... 127
F-1 Scatterplot of PSU Screener Response Rate by PSU Bachelor Degree (or Higher) Rates ... 137
F-2 Scatterplot of PSU Screener Response Rate by PSU Median Family Income.. ............ ...... 139
G-1 Percentages of Youths within Income Categories, CPS vs. NLSY ..... ............ ............ ...... 146
M-1 Number of Cases: Screeners and Non-Interviews.... ............ ............ ............ ............ ...... 183
M-2 Counting the Number of Youths from the Inhabited Households ...... ............ ............ ...... 195
R-1 Histogram of CX Non-Minority Youth Percentages . ............ ............ ............ ............ ...... 232
R-2 Histogram of CX All Youth Percentages...... ............ ............ ............ ............ ............ ...... 233
R-3 Histogram of SU All Youth Percentages ...... ............ ............ ............ ............ ............ ...... 233
R-4 Relationship of PSU Shortfalls to PSU Screener Response Rates – CX .......... ............ ...... 238
R-5 Relationship of PSU Shortfalls to PSU Screener Response Rates – SU........... ............ ...... 238
R-6 Relationship of PSU Shortfalls to PSU Education – CX ........ ............ ............ ............ ...... 240
R-7 Relationship of PSU Shortfalls to PSU Education – SU ........ ............ ............ ............ ...... 240
R-8 Relationship of PSU Shortfalls to PSU Median Family Income – CX ............ ............ ...... 241
R-9 Relationship of PSU Shortfalls to PSU Median Family Income – SU ............ ............ ...... 241
T-1 Weighted Percentages of Persons, NLSY97 Urban Blocks and CPS .. ............ ............ ...... 249
9
T-2 Weighted Percentages of Persons, NLSY97 Rural Blocks and CPS ... ............ ............ ...... 249
T-3 Weighted Percentages of Persons, NLSY97 Urban Central Place
Blocks and CPS... ............ ............ ............ ............ ............ ............ ............ ............ ...... 250
T-4 Weighted Percentages of Persons, NLSY97 Urban/Fringe Blocks and CPS.... ............ ...... 250
FT-5 Weighted Percentages of Persons, NLSY97 Urban,
Outside UA Blocks and CPS ....... ............ ............ ............ ............ ............ ............ ...... 251
10
Chapter 1. Introduction
The National Longitudinal Survey of Youth (NLSY97) is the latest in a series of surveys
sponsored by the U.S. Department of Labor (DOL), Bureau of Labor Statistics (BLS) to examine issues
surrounding youth entry into the work force and subsequent transitions in and out of the work force.
Originally done in 1979, the study followed a cohort of approximately 12,700 youths who completed a
round 1 baseline interview. The NLSY97 will follow a new cohort of approximately 9,000 youths who
completed an interview in 1997 (the base year).
The overall study design for NLSY97 incorporated the Profile of American Youth (PAY97)1 and
resulted in one large screening sample of over 90,000 housing units to generate youth participants for
both NLSY97 and PAY97. The NLSY97 sample was selected to represent the civilian, noninstitutional
population of the United States within the eligible age range – 12 to 16 years of age as of December 31,
1996 -- with oversamples of Hispanics and non-Hispanic blacks. Eligible youths in the screening sample
(and their parents) were administered a questionnaire that covered a range of topics, including work
experience, education, work-related attitudes, and other labor force and human capital issues.
This Technical Sampling Report gives detailed descriptions of the NLSY97 sample, including the
large screening sample. We also describe the estimation procedures employed to enable valid inferences
from the sample to the universe of all eligible American youths. Chapter 2 discusses the design of the
sample, and Chapter 3 the selection and implementation of the sample. Estimation procedures appear in
Chapter 4. In Chapter 5, we provide information about the potential impact of nonresponse on the study
results, the representativeness of the sample, and the precision of the survey estimators. We close the
report in Chapter 6 with a brief summary.
1
The PAY97, sponsored by the Department of Defense (DoD), is the second round of a large-scale social research
project designed to assess the vocational aptitudes of contemporary American youth and to establish current
national norms for the Armed Services Vocational Aptitude Battery (ASVAB) and for the Interest Finder (IF). This
project is in two parts; the Enlistment Testing Program (ETP) provided an assessment of roughly 6,000 youths aged
18-23, and the Student Testing Program (STP) provided an assessment of approximately 4,700 youths enrolled in
grades 10, 11, or 12 as of Fall 1997.
11
Chapter 2. Design of the NLSY97 Sample
2.1
The Eligible Population
In this section, we define exactly who was included in the universe for the NLSY97. Careful
definition of the universe is critical for purposes of sampling, designing and administering the screening
interview, and construction of weights.
The universe definitions are driven fundamentally by two concepts: (1) the housing unit (HU) and
(2) the usual place of residence. The Census Bureau provides the following definitions:
1. A housing unit is a single room or group of rooms intended as separate living quarters by a
family, by a group of unrelated persons living together, or by a person living alone. Separate
living quarters are those for which the occupant does not live or eat with other persons within the
structure and has direct access from the outside of the building or through a common hall which
is used or intended to be used by the occupants of another unit or the general public.
2. A person's usual place of residence is the place where the person lives and sleeps most of the
time.
These definitions prevailed throughout the NLSY97 project.
All housing units (HU's) within the land area of the 50 States and the District of Columbia were
considered within the in-scope universe of structures for NLSY97. The universe explicitly excluded all
group quarters structures, (e.g., prisons, college dormitories, military barracks, and nursing homes).
However, youths living in these group quarters, as specified below, were linked to an in-scope housing
unit.
All structures within the in-scope universe were subjected to area probability sampling. This
means that every HU within the universe had a known, nonzero probability of selection.
The sampling universe for the NLSY97 included
(a.1) persons aged 12-16 as of the reference day, December 31, 1996
who met at least one of the following criteria:
(r.1) persons whose usual place of residence was within the universe of structures set forth in
the previous paragraph
Including
•
persons who were temporarily away from their in-scope residence on vacation or in
a general hospital
12
•
persons with no other home who were staying at an in-scope HU at the time of the
screening interview
Excluding
•
persons who were visiting temporarily and usually live somewhere else, such as
overseas or in a mental hospital
(r.2)
children in a boarding school (both domestic and foreign schools) below the college level
(r.3)
college students in dormitories
(r.4)
persons in a jail, prison, or similar detention facility.
Youths in the latter three categories were included and linked to their parent's or guardian's HU,
provided that a parent or guardian was alive and identifiable and that the HU was within the in-scope
universe.
Youths who could be linked to two or more HU's (e.g., children of divorced or separated parents)
were linked to the mother's HU, provided the mother was alive and her HU was in-scope. Else, such
youths were linked to the father's HU, provided the father was alive and his HU was in-scope. Else, such
youths were linked to one of the linkable HU's at random.
All eligibility criteria were evaluated as of the time of the screening interview.
Eligible youths who screened into the sample were administered the NLSY97 questionnaire, the
ASVAB, and the Interest Finder (IF). The ASVAB and IF were administered under standardized
conditions at Sylvan Learning Centers.
Thus far, we have been discussing the sampling universe for the NLSY97. Yet sometimes a
survey's sampling universe is slightly different from its weighting universe, i.e., the universe to which one
plans to make inferences. Typically, the weighting universe may be a bit broader than the sampling
universe, reflecting the fact that some domains at the margin are extremely difficult or costly to survey.
For the NLSY97, the weighting universe consisted of all age-eligible youths in the civilian,
noninstitutional population of the U.S. Thus, the sampling and weighting universes were nearly identical,
but the weighting universe did include a few small categories not included in the sampling universe, such
as
•
homeless youths
•
youths living in noninstitutional group quarters who were not linkable to a household, e.g.,
youths living with their families in rooming houses or in agriculture workers' dormitories.
The sampling universe also included youths in a jail, prison, or a similar detention facility who were not
included in the weighting universe.
13
2.2
Overview of the Sample
The NLSY97 samples were selected in two broad phases. First, we specified a large sample of
more than 90,000 housing units. Through fieldwork, we determined which housing units were occupied
and which were vacant, and for the occupied housing units, we conducted brief screening interviews. In
the second phase, we selected subsamples of the eligible youths identified during screening for
participation in the main NLSY97 interview.
Table 2.1 presents the target samples sizes, overall and by demographic domain, we used for
planning the NLSY97. The table also shows the domain sample sizes we would achieve in simple
random samples of the same overall size. Clearly, the NLSY97 targeted sizeable oversamples of Hispanic
and non-Hispanic, black youths.
We designed the sampling methods and procedures to yield a database of eligible youths that
could be projected (with known confidence levels) to represent the entire eligible population of American
youths. For the large screening sample, our approach involved the selection of two independent areaprobability samples:
1. a cross-sectional (CX) sample designed to represent the various segments of the eligible
population in their proper population proportions, and
2. a supplemental (SU) sample designed to produce, in the most statistically efficient way, the
required oversamples of Hispanic and non-Hispanic, black youths.
Table 2.1 Target Sample Sizes for the NLSY97
Proportion of the
Eligible Population
Hypothetical Sample
Size in a Simple
Random Sample
Target
Sample Size
Total
1.000
10,000
10,000
Hispanic
0.129
1,290
1,667
Non-Hispanic Black
0.154
1,540
2,500
Non-Hispanic Nonblack
0.717
7,170
5,833
Domain
Source: Proportions are from the March 1997 Current Population Survey (CPS).
As an alternative, we could have used one area-probability sample large enough to meet the needs of both
the proportionate sample and the oversamples. After consideration, however, we decided to design one
area-probability sample to meet the needs of the proportionate sample, and a second area-probability
sample expressly to meet the needs of the oversamples. Our decision was based on the view that two
samples would provide greater value than one in terms of supplying precise statistical information at
lower overall cost.
14
Both the cross-sectional (CX) and supplemental (SU) samples were selected by standard areaprobability sampling methods.2 Sampling was in three essential stages: primary sampling units (PSUs),
segments, and HUs. NORC listed all addresses in the selected segments prior to the third stage of
sampling. The CX sample resulted from an essentially self-weighting design. Stratification was
introduced at the last stage for the SU sample to increase the hit rate for Hispanic and non-Hispanic, black
youths. A half-open interval procedure was used in both samples at the time of screening to include
persons living in housing units missed during the initial listing.
For the CX sample, we defined PSUs as Metropolitan Statistical Areas (MSAs) and as single
counties or clusters of neighboring counties in nonmetropolitan areas. For the SU sample, to enable
improved targeting of Hispanic and black youths, we defined PSUs as single counties. We selected 100
PSUs for each sample.
For both samples, we divided the selected PSUs into segments consisting of single census blocks
or clusters of neighboring blocks. Then, we selected 1,151 and 600 segments for the CX and SU samples,
respectively. In Fall 1995, NORC field workers visited the selected segments and listed all of the
addresses found therein. To guide their work, we prepared and provided field workers with block maps
drawn by Mapinfo software and driven by the Census Bureau’s TIGER database.
At the third and final stage of sampling, we selected addresses (presumed to be HUs) from the
NORC listing. For the SU sample, we classified the selected segments by high or low density (of
Hispanic or black youths in the 1990 Census) and sampled addresses in the high-density segments at 10
times the rate of sampling in the low-density segments.
Table 2.2 summarizes the sizes of the CX and SU samples. Overall, we originally selected
90,139 HUs from the NORC listing of 270,197 HUs.
Table 2.2 Sizes of the CX and SU Samples
Stage of Sampling
PSUs Selected
Segments Selected
Listed HUs
Selected HUs
Originally Selected HUs
HUs Obtained via Half-Open Interval
Procedure
2
CX Sample
SU Sample
Total Sample
100
100
200
1,151
600
1,751
176,673
93,524
270,197
65,269
25,688
90,957
64,654
25,485
90,139
615
203
818
This report assumes basic familiarity with these sampling methods. Readers in need of additional information on
area probability sampling should consult a standard survey sampling textbook.
15
Beginning in late January 1997 and continuing through October 1997, NORC conducted face-toface screening interviews at the selected HUs using a screen-and-go approach. During these field
operations, NORC field interviewers implemented a half-open interval procedure and identified and
screened an additional 818 HUs that were not included in the original listing.
Table 2.3 gives the results of the screening operation. From the last column on the right, it is
evident that we achieved an overall screener response rate of about 94 percent.
Table 2.3 Results of Screening Operations
Percent of
Eligible HHs
Eligible to be
Interviewed
Count of HUs
Percent of
All HUs
Type C Noninterview
2,139
2.4
Type B Noninterview
8,401
9.2
10,752
11.6
4,795
5.3
6.0
Completed Screener
75,410
82.9
94.0
SUBTOTAL
80,205
88.2
100.0
212
0.2
90,957
100.0
Status of HUs
Ineligible
TOTAL INELIGIBLES
Eligible to be Interviewed
Type A1 Noninterview
Other Eligible
Type A2 Noninterview
TOTAL OVERALL
NOTE: Type C noninterviews include businesses and demolished units. Type B noninterviews include
vacant HUs or HUs occupied by persons ineligible for the screening interview. Type A noninterviews
include HHs eligible for the screening interview for which the interviewer was unable to obtain a
complete interview. A2 includes HHs eligible for the survey but not eligible for the survey interview,
including “too ill or handicapped” and “language barrier.” A1 includes HHs eligible for the survey and
for the survey interview.
Figure 2.1 depicts our sampling and interviewing strategies for the NLSY97. For the CX sample,
all eligible youths identified in screening were designated for the main NLSY97 interview. For the SU
sample, however, only Hispanic and black youths were designated for the main interview. Non-Hispanic,
nonblack youths in the SU sample were screened out and not included in the NLSY97 sample of youths.
All youths designated for the interview were also designated for the ASVAB and IF tests, except for a
few cases coded as out-of-scope.
16
Figure 2.1 Map of CX and SU Samples for NLSY97
CX Sample
SU Sample
Hispanic
Non-Hispanic Black
Non-Hispanic, Nonblack
We included all eligible youths within each respondent household. Even in the case of
multiracial, multiethnic households, we kept whole households together. This decision actually resulted
in obtaining a few NLSY97 interviews from non-Hispanic, nonblack youths in the SU sample who lived
with eligible black or Hispanic youths. For certain technical reasons, however, we chose to disregard
these few cases in weighting the overall sample of non-Hispanic, nonblack youths. Each such youth was
finally assigned a zero weight.
Table 2.4 presents achieved sample sizes at the main interview and testing stages. For example,
9,808 age-eligible youths screened into the NLSY97, 8,984 completed an interview, and 7,168
participated in the ASVAB and IF testing. Differences between the number of youths identified in
screening and the number of youth participants are due to several factors, including classification of some
youths as out-of-scope, and nonparticipation including refusal. More details about these differences are
presented later.
Table 2.4 Achieved Sample Sizes for the NLSY97
Located in
Screening#
Subsampled for
Participation
Completed
Main
Interview*
Out-of-Scope
for Testing
Completed
Tests
Total
9,808
9,808
8,984
178
7,168
Hispanic
2,128
2,128
1,905
79
1,366
Non-Hispanic Black
2,546
2,546
2,333
45
1,807
Non-Hispanic Nonblack
5,134
5,134
4,746
54
3,921
Domain
#
These counts exclude non-Hispanic, nonblack youths, age 12-16, discovered in the SU sample.
Three non-Hispanic, nonblack individuals were screened into the supplemental sample and completed the
interview, but are excluded from the counts in this table and elsewhere in this volume. These individuals were
interviewed because they were siblings of eligibles, or else their race or ethnicity was unknown or misrepresented at
the time of the initial interview. NLSY97 databases may contain interview data for these cases, yet each has been
assgined an estimator weight of zero.
*
Response rates based on eligible youths can be determined with the aid of Table 2.4. Of the
17
eligible youths screened into the NLSY97 sample, 92 percent completed the youth interview and 73
percent participated in the ASVAB test. We present detailed information about participation rates in
Chapter 5.
2.3
Screening Questions and Forms
We designed and implemented two screening forms for NLSY97: a brief paper screener and an
extended CAPI screener. The paper form -- a copy of which is attached as Appendix H -- basically
sought to determine whether any residents were age eligible.
If age eligibles were reported, we then administered the extended screener, which sought more
explicit information about date of birth, age, other demographic characteristics, grade, and the like for
each resident (and linked person) of the household (HH). If no age eligibles were reported, then we did
not administer the extended screener and, by implication, did not collect demographic and other
information for HH members. A copy of the date of birth and age questions from the extended screener
appears in Appendix I.
In the foregoing paragraph, we described the general usage of the two screening forms. There
were exceptions where the paper form was omitted.
To administer the screener, interviewers went to the door of the housing unit and attempted to
gain the cooperation of a responsible adult who usually lived at the residence. Once cooperation was
gained, the interviewer administered the three minute household screener to determine if there were any
potentially eligible youths in the household. If the screener indicated the presence of such youths, the
interviewer attempted to complete the extended screener.
During a pretest for NLSY97, NORC and BLS realized that many households were extremely
costly to reach. Field interviewers made ten to twenty visits and still found no one home. Subsequently,
we decided to allow the interviewers to receive help from specified neighbors in determining whether
eligible youths lived at the sample households. Specifically, we decided to permit proxy and gatekeeper
interviews, under strict protocols, as described in Appendices E and L. Forms used by proxy and
gatekeeper respondents appear in Appendix J.
To achieve a high screener response rate, we also employed advance letters and a brochure, all of
which appear in Appendix K. The first advance letter, signed by the BLS project director Michael W.
Horrigan, was sent by first-class mail prior to the screening interview. It alerted the HH to the purpose of
the NLSY97, its importance, a small $10 incentive payment, sponsorship, and confidentiality.
The second letter, signed by NORC project director Harrison Greene, was handed to the HH
respondent at the time of the screening interview. This letter was specific to PAY97 and told of the $75
incentive payment for participation in the testing program.
The brochure, handed to the screener respondent at the screening interview, repeated much of the
information sent in the first advance letter. It also gave answers to some frequently asked questions.
18
Screening achieved considerable apparent success. During planning stages, for the overall
project, we assumed that 91 percent of occupied HUs would respond to the screening interview. In actual
practice, NORC obtained a 94 percent screener response rate. We present more details about screener,
interview, and test participation rates in Chapter 5.
19
Chapter 3. Selection of the NLSY97 Sample
In this chapter, we provide details of the procedures followed in selecting and implementing the
screening sample. We also describe the selection of youths for the NLSY97 sample.
3.1
Selection of the Cross-Sectional (CX) Sample
Selection of PSUs. We used NORC's 1990 National Sample for the CX PSUs. This sample
includes 100 PSUs selected to represent all 50 states and the District of Columbia. Each PSU is defined
as either a metropolitan area or one or more counties with a minimum size of 2,000 housing units (HUs).
Nineteen of the metropolitan PSUs contained such a large population they were included in the sample
with certainty; we will refer to these 19 as the certainty PSUs. Table 3.1 gives a tabulation of the PSUs
by type and region.
To increase the precision of sample estimates, the noncertainty PSUs were implicity stratified by
sorting them on several variables: whether the PSU was a metropolitan statistical area (MSA) or set of
non-MSA counties, state or census division, proportion of the 1990 population that was black or Hispanic
(grouped into quartiles), and per capita income. Sample PSUs were selected using a systematic sampling
scheme, with the selection probability for a PSU proportional to the number of housing units counted in
the 1990 Census. Therefore, the inclusion probability for the i-th PSU is given by
Βi
= 1
= n
, if i is one of the 19 certainty PSUs
Xi+
X++
, otherwise,
where n = 81 is the number of noncertainty selections, Xi+ is the 1990 count of HUs in the i-th PSU, and
X++ is the 1990 count of HUs across all noncertainty PSUs in the universe.
Table 3.1 Distribution of CX PSUs, by Census Region and Metropolitan Status
Census Regions
Total
Type of PSU
Northeast
Midwest
South
West
Certainty Metropolitan
4
4
6
5
19
Other Metropolitan
8
13
21
9
51
Nonmetropolitan
4
9
13
4
30
16
26
40
18
100
Total
20
Selection of Segments. The second stage sampling units, or segments, consisted of one or more
neighboring blocks within PSUs. Blocks were linked to assure that each segment included at least 75
HUs based on the relevant 1990 Census data. In total, we selected 1,151 segments within the 100 sample
PSUs -- with 9 or 10 in each of the noncertainty PSUs and up to 80 in the largest certainty PSU. For the
19 certainty PSUs (really strata from a technical point of view) and the one noncertainty stratum
consisting of all other areas, we allocated integer numbers of segments in rough proportion to the 1990
HU counts of the strata.
Prior to selection, we sorted the segments (implicity stratified) successively by (i) PSU, (ii)
whether or not they were in the central city (for metropolitan PSUs), (iii) state, (iv) county, (v) place
within county, (vi) percent minority quartile, and (vii) census tract. We sampled segments independently
from PSU to PSU using a systematic sampling scheme with probabilities proportional to the 1990 Census
count of housing units.
The selection probability for segment j within PSU i (conditional on the selection of the PSU) is
given by
π ij = mi
X ij
Xi+
,
where mi is the number of segments selected from the PSU and Xij is the 1990 count of HUs.
There was one segment in the selected PSUs so large that
X ij > X i + / mi .
We designated this segment as a certainty segment, included this segment in the sample and set π ij = 1 .
In the CX sample, 76 segments required additional subsampling prior to the selection of HUs for
screening. At NORC, we refer to this intermediate stage of sampling as chunking. The need for chunking
arises during a listing operation when field workers discover segments that have grown a lot since the
time of the original measure of size. We instructed the field workers to partition each of the 76 segments
based upon visible, replicable boundaries and to give a count of the HUs within each cell of the partition.
Subsequently, we selected one cell (or chunk) at random per segment, with probability proportional to the
field counts. For simplicity, we will continue to refer to the unit of sampling at the stage prior to the HU
stage as the "segment," even though for 76 of the segments there is actually an additional stage of
sampling. Henceforth, we shall also assume that the chunking probability has been incorporated in the
Βij.
Mapping and Listing the Selected Segments. Once the segments were selected, maps were
produced for each one using software that manipulates the Census Bureau's TIGER files. The maps
displayed the segment boundaries, roads, and other important features of the segment needed to manage
field work. NORC field staff then compiled a list of all the HUs within each sample segment.
21
Selection of HUs for Screening. The final stage of sampling involved the selection of HUs for
screening. We planned the CX screening sample so that it would produce the 5,833 interviews with nonHispanic, nonblack youths targeted by the NLSY97. The same sample, of course, would provide
proportional representation of Hispanic and non-Hispanic, black youths. We planned the SU screening
sample so that it would be large enough to produce the oversamples of Hispanic and non-Hispanic, black
youths.
In light of the foregoing plans, 5,833 interviews with non-Hispanic, nonblack youths became the
driver for the CX screening sample size. We estimated the number of HUs needed to be screened to
achieve this goal:
(1)
Based on data from the March 1993 Current Population Survey (CPS), we estimated that
for each occupied housing unit in the US, there would be .131 non-Hispanic, nonblack
youths between the ages of 12 and 16 years old;
(2)
Based on experience in recent surveys, we assumed that approximately 85 percent of the
HUs fielded would be occupied;
(3)
We assumed that, as in the NLSY79, screening interviews would be completed at 91
percent of these occupied HUs; and
(4)
We assumed that, as for NLSY79, 89 percent of youths selected for the sample would
complete the NLSY97 interview.
Given, these conditions, we selected and subsequently screened 64,654 HUs (approximately 5,833 over
the product of .131 × .85 × .91 × .89) for the CX sample.
To achieve a self-weighting sample of HUs, we determined that the conditional sampling rate for
HUs k in segment (i, j) should be
π ijk = f o / (π iπ ij ) ,
where fo = qo / Q is the overall sampling rate for HUs, qo = 64,654 is the sample size for HUs, and Q is an
estimate of the nation's housing stock. The within-segment sampling rate was constant across all housing
units (subscripted by k) within the segment. This method of sampling caused the final cluster sizes within
the CX sample to be roughly constant, varying only due to any discrepancies between the 1990 Census
figures and the actual number of listed HUs.
In the cross-sectional sample, three segments had a calculated Βijk > 1. For these segments, Βijk
was limited to 1 (i.e., the selection of all listed housing units within these segments). It should be pointed
out that the reason for Βijk to be calculated as greater than 1 is that the probability of selection for the
segment is already smaller than the desired equalized probability for housing units. Even by choosing
every housing unit in these segments, the selection probability for a housing unit cannot exceed the
selection probability for its segment.
22
We sampled HUs using systematic sampling, according to the rates Βijk, using the order of HUs
determined by the NORC listing. Sampling was independent from segment to segment.
In Table 3.2, we present the sample sizes calculated for the CX sample during the planning stages
of the project. The difference between the actual number of HUs selected and fielded, 64,654, and the
number, presented in this table, 64,684, is due to rounding. At the planning stage of the project, it looked
like the CX sample would supply 9,001 youths aged 12-16. The SU sample would be designed to supply
the remaining black and Hispanic youths needed for the NLSY97.
Screening for Eligibility. NORC field staff conducted screening interviews at the specified HUs
to find out the number of persons residing there, as well as their ages, races/ethnicities, grades, genders,
and other eligibility information. The answers to these questions determined which, if any, household
members were eligible for the main interview.
In the course of the screening operation, we executed the half-open interval (HOI) procedure to
detect and screen both HUs missed by the original listing operation and HUs resulting from new
construction. The HOI procedure is standard practice at NORC for area probability samples. A detailed
description is available upon request. If one, two, or three new HUs were discovered at an original
selection, then each was added to the screening sample. If four or more new HUs were discovered at an
original selection, then we specified a systematic sample of three of the new HUs for the screening
sample and incorporated the sampling probability into the corresponding Βijk.
Table 3.2 Projected Screening Sample Sizes and Yields Used in Planning the NLSY97
Sampling Units
HUs Selected
CX Sample
Number
64,684
Rate
–
SU Sample
Number
31,753
Rate
–
Total
Number
96,437
Occupied HUs
54,981
0.8500
26,990
0.8500
81,971
Completed Screeners
50,033
0.9100
24,561
0.9100
74,594
Total Person Rostered
131,086
2.6200
64,350
2.6200
195,436
9,001
0.0687
4,419
0.0687
13,420
Hispanics
1,027
0.0078
1,008
0.0157
2,035
Non-Hispanic Blacks
1,420
0.0108
1,389
0.0216
2,809
Non-Hispanic Nonblacks
6,554
0.0500
2,022
0.0314
8,576
NLSY97 Youth Interviews
8,011
0.8900
2,133
0.8900
10,144
914
0.8900
897
0.8900
1,811
Non-Hispanic Blacks
1,264
0.8900
1,236
0.8900
2,500
Non-Hispanic Nonblacks
5,833
0.8900
–
–
5,833
Persons Aged 12-16
Hispanics
23
Selecting Youths for Participation in NLSY97. The actual numbers of eligible youths
identified in the screening operation appear in Table 3.3. Overall, 9,808 NLSY97-eligible youths
screened in. These actual counts are notably smaller than the projected counts from Table 3.2. We will
return to this disparity later. But first, we consider the SU sample.
Table 3.3 Actual Screening Sample Sizes
Youths
CX Sample Number
SU Sample Number
Total Number
7,335
2,473
9,808
Hispanics
1,026
1,102
2,128
Non-Hispanic Blacks
1,175
1,371
2,546
Non-Hispanic Nonblacks
5,134
0
5,134
NLSY97-Eligible Persons
3.2
Selection of the Supplemental (SU) Sample
Selection of PSUs. The SU screening sample was selected using procedures very similar to those
used to select the CX sample, but with the intention of oversampling black and Hispanic youths. The
PSUs were defined differently from those for the CX sample: all PSUs for the SU sample were single
counties, not clusters of counties or MSAs. We chose not to use MSAs because of our goal to oversample
black and Hispanic youths; MSAs often have counties with widely differing minority rates. Thus, it made
sense to separate counties with higher percentages of minorities from those with lower percentages. The
easiest way to accomplish this was simply to use counties as the PSUs.
We merged counties into PSUs to meet a minimum size criterion of 2,000 HUs. To guarantee
that the PSUs with the highest minority youth rates were properly represented, we implicitly stratified the
frame by sorting PSUs first by minority youth density (number of blacks and Hispanics, aged 17 and
under, per housing unit) grouped into thirds. Within each third, we sorted by region, division,
metropolitan status, state, and per capita income. Then, we selected a systematic sample of 100 PSUs,
with the selection probability for each PSU proportional to its measure of size.
We used as the PSU measure of size the weighted number of black and Hispanic youths aged 17
and younger in the PSU:
X i + = max (1, 1000 f H N Hi + 1000 f B N Bi )
where fH and fB refer to the overall sampling rates for Hispanic and non-Hispanic, black youths in the
NLSY97-eligible age range, and NHi and NBi refer to 1990 Census counts of Hispanic and black youths
within the PSU. The sampling rates are shown in Table 3.4. As it happened, the sampling rates for the
two groups were similar, and thus the measure of size was essentially proportional to the simple sum of
blacks and Hispanics in the eligible age range within the PSU. For the NHi and NBi, we would have
preferred to use nonoverlapping race/ethnicity categories, such as Hispanic and non-Hispanic black, but
the relevant 1990 Census files did not provide separate counts at the county level of persons by age, race,
24
and Hispanic origin.3 Instead, we decided to use the overlapping categories black and Hispanic. This
treatment has little statistical consequence, because very few Hispanics (less than 3.5 percent of the total)
classify themselves as black in the decennial census.
Table 3.4 Sample Sizes and Sampling Rates Used in Determining
the Measure of Size for the SU Sample
Domain
Hispanic
Non-Hispanic Black
All Others
2,000
3,000
7,000
.000822
.000896
.000451
Age 12-17
Target Sample Size
Sampling Rate
The sampling rates in Table 3.4 are based on a total sample size of 12,000 youths aged 12-17,
which were the figures contemplated at the time of sample implementation. Later, just before NLSY97
went into the field, in an effort to reduce cost, the BLS cut the age range to 12-16 and the sample size to
10,000. Notwithstanding the cut, Table 3.4 reflects the fH and fB we actually used in defining the measure
of size.
The selected sample of 100 PSUs appears in Table 3.5, broken down by region, certainty, and
metropolitan status. All 16 nonmetropolitan PSUs were noncertainty PSUs, and all 17 certainty PSUs
were metropolitan PSUs. Clearly, the overall SU sample is heavily concentrated in the South and the
West, reflecting well-known concentration patterns of the black and Hispanic populations.
Table 3.5 Distribution of SU PSUs, by Census Region and Metropolitan Status
Census Regions
Type of PSU
Total
Northeast
5
Midwest
2
South
5
West
5
17
Other Metropolitan
9
12
33
13
67
Nonmetropolitan
2
0
12
2
16
16
14
50
20
100
Certainty Metropolitan
Total
3
In defining the measure of size, we worked with county-level data from the 1990 Census, specifically the STF 1C
data.
25
Selection of Segments. At the next stage of sampling, we selected a total of 600 segments from
a sorted list of all the segments within the selected PSUs. In defining segments, census blocks were
collapsed to reach a minimum size of 75 HUs. The measure of size was defined by
X ij = max (1, 1000 f H N Hij + 1000 f B N Bij )
and the conditional inclusion probability by
π ij = mi
X ij
Xi+
,
for the j-th segment in the i-th PSU, where NHij and NBij refer to 1990 counts of Hispanic and nonHispanic black youths, respectively, aged 0-17.4
Prior to sampling, the list of segments was sorted by (i) PSU (i.e., county), (ii) minority youth
density grouped in thirds, (iii) percentage of minority youths who were Hispanic (grouped into thirds),
(iv) place within county, (v) percent minority quartile, and (vi) census tract. Note that sort (iii) was done
to guarantee that the sample of segments would be representative of both minority groups (blacks and
Hispanics). Sampling was independent from PSU to PSU.
Mapping and Listing of the Selected Segments. Just as for the CX sample, once the SU
segments were selected, maps were produced for each one. NORC field staff then compiled a list of all
the HUs within each sample segment.
Selection of HUs for Screening. We observe, once again, that the screening sample size was
driven by the needs of the NLSY97. Because the sampling rate for non-Hispanic blacks was highest
among the three targeted race/ethnicity domains (see Table 3.4), we designed the SU screening sample to
provide us with enough eligible non-Hispanic blacks to achieve 2,500 completed youth interviews. Such
a sample would be large enough, we figured, to achieve the targeted 1,667 Hispanic interviews. Using
the numbers from the CX sample given in Table 3.2, we determined that the SU sample must screen in an
additional 1,389 eligible non-Hispanic blacks aged 12 to 16. From the March 1993 CPS, we expected
there to be .057 such youths per household. Using the same rates of occupancy (85%) and of screener
response (91%), we planned to sample and screen about 31,753 HUs (approximately 1,389 over the
product of .057 × .85 × .91). With an assumed 89 percent response rate, the planned screening sample of
2,809 non-Hispanic, black youths would yield exactly 2,500 completed interviews as targeted.
Table 3.2 lays out all of the sample sizes calculated for the SU sample during the planning stage
of the overall project. There, we see that the SU sample was projected to yield 897 completed Hispanic
interviews, more than enough to achieve the overall target of 1,667 completed Hispanic interviews.
4
In defining the measure of size, we worked with county-level data from the 1990 Census, specially the STF 1C
data.
26
As screening operations were launched for NLSY97/PAY97, cost considerations came into play,
and NORC and the BLS decided to release only about 80 percent of the planned SU sample of 31,753
HUs, reserving the balance for a later possible release. Thus, we released an SU screening sample of
exactly 25,485 HUs. Our thinking was that money could be saved by releasing the smaller sample. Our
plan, at that time, was to monitor the yields of youths as screening interviews proceeded, and to release
the reserved HUs in the event that yields were falling short of targets. Ultimately, we never did release
the reserve, despite an early and obvious shortfall in the number of youths.
To achieve a self-weighting sample of HUs, we determined that the conditional sampling rate for
HUs k in segment (i, j) should be
π ijk = f 1 / (π iπ ij ) ,
where f 1 = q1 / Q is the overall sampling rate for HUs, q1 = 31,753 is the sample size for HUs, and Q is a
current estimate of the nation's housing stock. Rather than a self-weighting design, however, we decided
to oversample HUs within segments with large concentrations of black or Hispanic youths by creating
two classes of segments based on the segment-specific rates of minority youths in the 1990 Census. A
segment was classified as “high-minority” if either the rate of black youths under age 18 per housing unit
or the rate of Hispanic youths under age 18 per housing unit was at least 0.20 (i.e., one youth for every
five housing units). Otherwise, a segment was classified as “low-minority” (even if the sum of the two
rates was at least 0.20).
We determined two different sampling rates f high and f low for the two classes. The proper ratio
between f high and f low was a key decision. Increasing the ratio would increase the sampling efficiency
in targeting minority youths, but would also increase the variability in final sampling weights. The ratio
chosen was 10:1. This means that housing units in “high minority” segments would have a final
probability of selection 10 times that of housing units in “low minority” segments. The application of this
decision resulted in separate sampling rates for “high minority” and “low minority” segments, with
π ijk = f high / (π i π ij ) , if (i, j) is in the “high minority” class;
= f low / (π i π ij ) , if (i, j) is in the “low minority” class.
Sampling of HUs was independent from segment to segment.
Unlike the CX sample, in which only three segments had a calculated π ijk > 1, there were many
such segments in the SU sample, especially among the “high-minority” segments. As in the CX sample,
π ijk was limited to 1 (i.e., all listed housing units were selected).
27
Table 3.6 summaries the key features of the actual CX and SU screening samples.
Table 3.6 Summary of the Design of the CX and SU Samples
Design
CX Sample
SU Sample
Number of Selections
100 PSUs
100 PSUs
Measure of Size (MOS)
Housing units (HUs)
Minimum Size
2,000 HUs
Weighted sum of black and
Hispanic youths, aged 17 and
below
2,000 HUs
Method of Selection
Systematic selection with
probabilities proportional to
size (pps)
Metropolitan status,
division/state, percent minority
quartile, per capita income
Systematic pps selection
Number of Selections
1,151
600
Measure of Size
HUs
Minimum Size
75 HUs
Weighted sum of black and
Hispanic youths aged 17 and
below
75 HUs
Method of Selection
Systematic pps selection
Systematic pps selection
Sort (implicit stratification)
PSU, central city, state, county,
place, percent minority
quartile, census tract/BNA
PSU, minority youth density
thirds, percentage of minority
youths who are Hispanic,
place within county, percent
minority quartile, census
tract/BNA
64,654
25,485
First Stage
Sort (implicit stratification)
Minority youth density thirds,
region, division, metropolitan
status, state, per capita income
Second Stage
Third Stage
Actual Number of HUs
Screening for Eligibility. NORC field staff conducted screening interviews at the selected HUs
to find out the number of persons residing there, as well as their ages, races/ethnicities, grades, genders,
and other eligibility information. The answers determined which, if any, household members were
eligible for the main interview.
Selecting Youths for Participation in NLSY97. The actual numbers of eligible youths located
in the screening operation appear in Table 3.3. Overall, 9,808 eligible youths screened in, and we
attempted to complete a main interview with each.
28
Chapter 4. Weighting the Sample
Data from large-scale national samples typically need to be weighted to achieve an unbiased
estimator of the population total. The weights are needed for four main reasons. First, the weights
compensate for differences in the selection probabilities of individual cases, which often arise by design,
as in the NLSY97/PAY97, where different overall sampling rates were required for Hispanics, nonHispanic blacks, and others within the eligible age ranges. Second, weighting compensates for subgroup
differences in participation rates; even if the sample as selected were representative of the larger
population, differences in participation rates can compromise the representativeness of the sample. For
example, different geographic areas may experience different rates of screener nonresponse. Such
differences in participation rates can introduce nonresponse bias into the results; weighting can reduce
these biases. Third, weights compensate for random fluctuations from known population totals due to
sampling. For instance, if one sex were overrepresented in the NLSY97 sample purely by chance, it
would be possible to use data from the Decennial Census or the Current Population Survey to adjust for
this departure from the population distribution. And fourth, adjusting the data to known population totals
can help reduce the impact of survey undercoverage (such as undercoverage arising from the omission of
persons in partially enumerated households).
The weights for the NLSY97 were computed in six steps:
Step 1. Computation of the base weight, reflecting the housing unit's selection probability for the
screening sample.
Step 2. Adjustment for household nonresponse to the screener.
Step 3. Adjustment to reflect subsampling of youths in screened households.
Step 4. Development of a combination weight to allow youths from the CX and SU samples to
be merged into one combined sample.
Step 5. Adjustment for youth nonresponse to the main interview.
Step 6. Poststratification of the nonresponse-adjusted weights.
This approach implicitly treats the CX and SU screening samples as stand-alone samples, which are then
combined in the fourth step. This approach was taken in NLSY79, and its soundness was attested to at
that time by a panel of experts including Joseph Sedransk, Leslie Kish, and Benjamin King (see Frankel
& McWilliams, 1981).
Step 1: Computation of the base weight. The base weight ( W1 ) for the k-th housing unit in the
screening sample is the inverse of the probability of selecting the unit:
W1k =
1
πk
,
where Βk denotes the relevant inclusion probability. For the CX sample, the base weights are mostly
identical because the sampling was designed to equalize the probability of selection for each housing unit.
However, as described in Chapter 3, three segments (out of 1,151) were selected with certainty, and
29
therefore, the housing units within these segments have a slightly larger base weight, and all other
housing units have a slightly lower base weight.
In the SU sample, segment probabilities were often smaller than the desired equalized
probabilities, especially in the segments “low” in minority youths (see Chapter 3). This resulted in larger
base weights for housing units within these segments. Segments “low” in minority youths already had an
equalized base weight 10 times larger than segments “high” in minority youths, and for some “low”
segments, the base weight threatens to be 70 times larger. To prevent extreme weights, we limited the
base weight to 2.0 × 9993.74 (the equalized base weight for “low” segments). It should be noted that this
cap reduces the sum of the weights in the SU sample by over 25 percent (see Section 5.2 for more details
on the effect of the weighting caps), which also reduces the coverage ratios. This is the price for the
efficiency of the supplemental sample, which did an excellent job of reducing the fieldwork necessary to
obtain the black and Hispanic oversamples.
There is one additional complication to this step. NORC utilized the half-open interval
procedure, as explained in Chapter 3. Since this procedure subsamples missed housing units, an
adjustment is necessary. This adjustment is simply multiplication by the number of missed housing units
found divided by the number selected. For example, if nine are found (three are always selected if there
are more than three), then each of the three selected have their base weight multiplied by 9/3 = 3 (i.e.
these three housing units represent the nine found in the sample). Unfortunately, this adjustment also has
the risk of causing extreme weights. In particular, one segment had 334 missed housing units attached to a
single sampled housing unit (the adjustment factor would be 111.33). To combat potentially extreme
weights, we truncated the adjustment to 5.0. This cap reduced the sum of the weights by only 0.40
percent (see Section 5.2 for more details on the effect of the weighting caps). The following table lists the
unadjusted and capped missed housing unit adjustments, and how many housing units had the adjustment
made to their weights.
30
Table 4.1 Unadjusted and Capped Missed Housing Unit Adjustments
Unadjusted Factor
Capped Factor
Number of HUs
1.00
1.33
1.67
2.00
2.33
2.67
3.00
3.67
4.33
5.00
6.67
7.33
10.00
13.00
15.00
16.33
16.67
111.33
Total
1.00
1.33
1.67
2.00
2.33
2.67
3.00
3.67
4.33
5.00
5.00
5.00
5.00
5.00
5.00
5.00
5.00
5.00
702
24
21
12
12
3
5
3
9
3
3
3
3
3
3
3
3
3
818
Step 2: Adjustment for screener nonresponse. The next step was to adjust the screener base
weights for nonresponse to the screening interviews. The nonresponse-adjusted weight ( W2 ) is the base
weight ( W1 ) inflated by the inverse of the weighted response rate within an adjustment cell:
W2 k =
∑W
j ∈Eα
1j
∑W
j ∈Sα
W1k ,
1j
where Eα is the set of eligible units within the α-th cell and Sα is the set of screener respondents in the α-th
cell. Because few variables, if any, are available for the screener nonrespondents, we based the
adjustment cells on simple geographic criteria. The cells were defined to be segments. Unfortunately,
some segments were too small to have a reliable nonresponse weighting adjustment. If there were fewer
than 25 respondents in a segment, we instead used the nonresponse weighting adjustment for the entire
PSU. Using the PSU nonresponse rate did prevent some large nonresponse adjustments, but the effect on
the sum of weights was less than 0.50 percent (see Section 5.2 for more details on the effect of various
weighting “caps”).
Four PSUs, however, had weighted PSU response rates that were considered too low to be used
for adjustment. Within these four PSUs, the nonresponse adjustment was capped at 2.0 to prevent extreme
weights. This is equivalent to saying that if the weighted segment (PSU if fewer than 25 respondents in
31
the segment) response rate was less than 50 percent, it was set to 50 percent for purposes of the weight
adjustment. This cap reduced the sum of the weights by only 0.22 percent in the SU sample (see Section
5.2 for more details on the effects of the weighting caps).
W2 represents the final screening sample weight and the final household weight.
Step 3: Adjustment for subsampling youths. We now turn to youth weights. Because eligible
youths were not subsampled for the youth interview, the initial youth weight is given by
W3k = W2 k ,
for the k-th youth, where W2 k is the step-2 weight assigned to the youth’s household. This youth
weighting derives from the fact that the youth’s probability of selection is equal to the youth’s
household’s probability of selection.
Step 4: Combining the cross-sectional and supplemental samples. Thus far, the weights have
treated the CX and SU sreening samples as stand-alone samples. The next step was to adjust the weights
from step 3 so that the two samples could be used together. In effect, we created a “precision” weight, λ ,
to combine estimators from the two samples:
θ$ = λθ$c + (1 − λ )θ$s
in which θ$c represents a statistic derived from the CX sample and θ$s represents the corresponding
statistic from the supplemental sample. Because the two samples are independent, the optimum λ is
proportional to the effective sample size in the CX sample:
λ=
1− λ =
nc / dc
nc / dc + ns / ds
ns / d s
nc / d c + ns / d s
,
where nc and ns are the nominal sample sizes for the CX and SU samples and d c and d s represent the
design effects for the estimators from each sample. It is inconvenient to use the design effects
themselves, since they will vary from one variable to the next. Instead, we used one plus the relative
variance of the weights within each sample (as was done for NLSY79); this factor captures the impact of
unequal weighting on sample efficiency:
λ$ =
nc / d$c
nc / d$c + ns / d$s
1 − λ$ =
ns / d$s
nc / d$c + ns / d$s
32
s(W3i , i ∈ CX)
d$c = 1 +
M(W3i , i ∈ CX)
2
s(W3i , i ∈ SU)
d$s = 1 +
M(W3i , i ∈ SU)
2
,
where M(.) and s(.) signify the simple mean and standard deviation of the weights, respectively.
The merging of the samples was done separately for each race/ethnicity by sex domain (e.g.,
Hispanic males). Race/ethnicity was defined as Hispanic, non-Hispanic black, or non-Hispanic, nonblack.
At the level of individual members of the sample, the weight at this stage ( W4 ) is the sample member's
weight from the previous step times the relevant combination factor ( λ$β ):
W4 k = λ$β W3k
i ∈ CX
= (1 − λ$β )W3k i ∈ SU ,
where ∃ signifies the sex by race/ethnicity domain. Shown below is a table listing the λ$s that were used
during the weight calculations.
Table 4.2 λs Used for Precision Weighting to Combine the
CX and SU Samples
Domain
Hispanic
Non-Hispanic, Black
All Other
Male
Female
Male
Female
Male
Female
CX (8)
.75
.75
.68
.66
1.00
1.00
SU (1-8)
.25
.25
.32
.34
.00
.00
It is clear that the CX cases are being given more weight for the minority categories; three-quarters for
Hispanics, and two-thirds for non-Hispanic blacks. This is because there is more variability in the
weights in the SU sample (caused by the oversampling for minority youths), reducing the effective
sample size. It is important to remember than non-minorities in the SU sample were ineligible for the
NLSY97 interview.
33
Step 5: Adjusting for nonresponse in the main data collection. For each sample, we adjusted the
weights computed in the third step for nonresponse in the main data collection. We used a fairly simple
scheme to create nonresponse adjustment cells. Cells were based on screening sample (CX, SU),
race/ethnicity (Hispanic, non-Hispanic black, and all other youths), age, and sex. The adjusted weight
( W5 ) is just the weight from Step 4 inflated by the inverse of the weighted response rate, with separate
adjustments computed for each cell. This step does not affect the sum of the weights. The only change is
the redistribution of the nonrespondents’ weights to the respondents. No combining of cells was required.
Also, no caps on the nonresponse weighting adjustment were necessary or used.
Table 4.3 shows the cell response rates used for the nonresponse weighting adjustment.
Table 4.3 Cell (Weighted) Response Rates (in Percent) Used for the
NLSY97 Nonresponse Weighting Adjustment
Domain
CX Sample
Hispanic
Male
Female
SU Sample
Male
Female
Age 12
Age 13
Age 14
Age 15
Age 16
Age 12
Age 13
Age 14
Age 15
Age 16
Age 12
Age 13
Age 14
Age 15
Age 16
Age 12
Age 13
Age 14
Age 15
Age 16
88.87
90.95
93.67
90.51
82.77
89.90
91.27
91.67
97.01
82.13
86.90
86.35
91.45
84.61
85.05
89.25
91.81
94.67
91.57
85.96
34
Non-Hispanic
Black
91.57
90.12
92.35
86.60
90.02
95.46
92.21
90.97
96.51
93.00
94.25
94.94
92.68
94.76
89.83
88.62
92.94
94.90
92.55
92.11
All Other
94.20
93.30
90.50
92.74
89.88
92.54
91.69
92.68
94.32
91.52
Not applicable
Not applicable
Not applicable
Not applicable
Not applicable
Not applicable
Not applicable
Not applicable
Not applicable
Not applicable
Step 6: Poststratifying the weights. The last step in the computation of the weights was to
adjust the weights to bring them into agreement with independent estimates of the population size:
W6 k = W5k ×
Tγ
.
∑W
5k ′
k′
For each cell (or poststratum), γ, the adjustment factor, is the ratio between an (independent and presumed
superior) estimate of the size of the population (Tγ) and the sum of the weights for sample members in that
cell. Cells were defined in terms of age, race/ethnicity, and sex. Benchmarks for the poststratification
adjustments were obtained from the March 1996 and 1997 Current Population Surveys (CPS), excluding
persons outside the household population, as in Table 4.4 below.
Table 4.4 Population Totals Used for the NLSY97 Poststratification
Hispanic
12-Year-Olds
13-Year-Olds
14-Year-Olds
15-Year-Olds
16-Year-Olds
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
249,654
236,321
250,174
233,112
269,949
243,954
295,675
216,934
271,631
225,988
Non-Hispanic
Black
301,005
291,417
298,899
300,464
314,067
293,977
293,540
299,701
316,369
272,632
All Other
1,483,161
1,365,341
1,354,138
1,331,628
1,426,387
1,376,555
1,385,961
1,370,011
1,432,737
1,377,071
The ratio of the sum of the sample weights after Step 5 ( W5 ) for a specific domain to the corresponding
CPS estimate of the size of that domain (i.e., the reciprocal of the poststratification factor) is one possible
estimate of the coverage of the NLSY97. Table 4.5 presents these coverage ratios.
35
Table 4.5 Coverage Ratios for NLSY97 Sample
12-Year-Olds
13-Year-Olds
14-Year-Olds
15-Year-Olds
16-Year-Olds
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Hispanic
.75
.77
.72
.75
.62
.81
.65
.85
.69
.71
Non-Hispanic Black
.62
.70
.68
.68
.74
.64
.69
.63
.65
.81
All Other
.62
.62
.71
.67
.67
.64
.68
.69
.62
.57
This table suggests that the coverage rate was about two-thirds (67 percent). In our judgement, however,
these ratios are not the best estimates of the coverage of the NLSY97 because they confound the coverage
effect and the effect of capping weights, which is intended to limit extremes and reduce sampling
variabilility. We analyze this matter again in Section 5.2, where we present our best estimates of
coverage and compare the coverage of NLSY97 to the coverage of the 1990 Census and the CPS.
We close this chapter with some summary tables for the NLSY97 intermediate and final weights.
Table 4.6 Summary Table of Housing Unit and Household Weights
W1
90,957
200,471,047
2,204.02
2,892.12
1,000.24
1,000.24
1,652.35
1,652.35
1,652.35
9,992.01
98,937.13
N (> 0)
Sum
Mean
Standard Deviation
Minimum (> 0)
5th percentile
25th percentile
Median
75th percentile
95th percentile
Maximum
36
W2
75,410
177,917,390
2,359.33
3,042.33
1,000.24
1,010.45
1,652.35
1,699.56
1,784.53
10,120.11
102,765.79
Table 4.7 Summary Table for Youth Weights
N (> 0)
Sum
Mean
Standard Deviation
Minimum (> 0)
5th percentile
25th percentile
Median
75th percentile
95th percentile
Maximum
W3
9,808
16,619,189
1,694.45
1,037.86
1,000.24
1,004.02
1,652.35
1,697.00
1,768.71
1,996.69
22,385.97
W4
9,808
12,858,858
1,311.06
635.56
248.88
264.09
1,089.75
1,652.35
1,731.03
1,893.44
9,056.13
37
W5
8,984
12,858,858
1,431.31
683.26
264.68
298.39
1,170.47
1,782.82
1,881.93
2,052.48
9,785.96
W6
8,984
19,378,453
2,157.00
1,070.78
323.30
420.26
1,518.77
2,601.05
2,922.39
3,298.97
15,759.42
Chapter 5. Accuracy and Precision of the Sample Results
5.1
Potential Impact of Nonresponse
Given the typical rates of completion currently obtained in U.S. household surveys, completion
rates for the various stages of the NLSY97 are fairly high. In what follows, we examine noncooperation at
the various stages of the survey process and discuss its potential impact on the survey results.
Participation Rates. Participation will be examined at several stages of the study, including
screening, interviewing, and testing. The screening process is the first opportunity for participation in the
NLSY97; final response rates for all screeners are given in Table 5.1. Among the ineligibles are Type C
noninterviews, consisting of condemned and demolished residences as well as businesses and nonexistent
housing units, and Type B noninterviews, which include group quarters, vacation homes, properties under
construction, and vacant housing units. Type A2 includes HHs eligible for the survey but not eligible for
the survey interview, including “too ill or handicapped” and “language barrier.” The eligibles for the
interview are made up of Type A1 noninterviews (refusals, breakoffs, and computer crashes) and of
completed screeners. As Table 5.1 shows, eligibles for the interview make up 88.2 percent of the total
screeners, among which 94.0 percent completed a screener. Among all cases, completions were 82.9
percent. Apparently, the screener response rate is slightly higher in the CX sample than in the SU sample.
Table 5.1 Final Response Status for All Screeners
Sample
CX
SU
Count
Percent
of SU
Cases
Count
Both Samples
Percent of
HHs Eligible
Percent
to be
of Cases Interviewed
Status of HUs
Ineligible
Type C Noninterview
Type B Noninterview
Count
Percent
of CX
Cases
1,427
6,146
2.19
9.42
712
2,255
2.77
8.78
2,139
8,401
2.35
9.24
Total Ineligibles
7,573
11.61
29,067
11.55
10,540
11.59
Type A1 Noninterview
Completed Screener
3,309
54,253
5.07
83.12
1,486
21,157
5.79
82.36
4,795
75,410
5.27
82.91
5.98
94.02
Subtotal
57,562
88.19
22,643
88.15
80,205
88.18
100.00
134
0.20
78
0.30
212
0.23
Eligible to be Interviewed
Other Eligible
Type A2 Nonresponse
65,269
100.00 25,688
100.00 90,957
100.00
Total Overall
Proxy and gatekeeper screeners make up a portion of the completed screeners (proxies and
gatekeepers were first discussed at the end of Chapter 2; see also Appendices E and L for reviews of
38
gatekeeper and proxy interviews); their rates, along with final screener completion rates, are given in
Table 5.2 below. Approximately 11.5 percent of the total eligible screeners and 12.2 percent of the total
completed screeners were proxy or gatekeeper interviews.
Table 5.2 Screener Completion Rates
Disposition
Proxy Screeners
Gatekeeper Screeners
Other Completes
Total Completed Screeners
Total Sample
Percent of Eligible
Screeners
6.45
5.06
82.51
94.02
Count
5,175
4,055
66,180
75,410
Percent of Completed
Screeners
6.86
5.38
87.76
100.00
Response rates for the NLSY97 interview and test are shown in Tables 5.3 and 5.4. For the
NLSY97 interview, ineligible youths are defined as those cases that, subsequent to the screening
interview, were determined to be age ineligible on the basis of updated age reports in the youth interview
or other detailed analyses of screening information. Table 5.3 indicates that 1.5 percent of all cases that
were thought to be NLSY97-eligible at the time of screening were, in fact, not eligible to participate in
the NLSY97 interview; an additional fraction of a percent were determined to be language barrier cases or
to be too ill or handicapped to participate. About 92.2 percent of the remaining eligible cases completed
the interview, while 90.3 percent of all cases completed an interview.
Table 5.3 Final Response Status for NLSY97 Youths
Status of Youths
Ineligible
Eligible to be Interviewed
Type A1 Nonresponse
Completed Interview
CX
Percent
of CX
Count
Cases
100
1.34
Sample
SU
Percent
of SU
Count
Cases Count
45
1.79
145
Both Samples
Percent
of Cases
1.46
Percent of
Eligibles
541
6,748
7.28
90.76
218
2,236
8.66
88.80
759
8,984
7.62
90.26
7.79
92.21
Subtotal
Other Eligible
Type A2 Nonresponse
7,289
98.04
2,454
97.46
9,943
97.89
100.00
46
0.62
19
0.75
65
0.65
Total Overall
7,435
100.00
2,518
100.00
9,953
100.00
Those youths who completed an NLSY97 interview were then asked to take the ASVAB test.
Table 5.4 displays unconditional participation rates for the test as well as conditional participation rates
given that an interview was completed. Noninterviews are made up of ineligibles and Type A1 and A2
39
nonrespondents, as defined earlier. Interviews also include out-of-scope youths, which are defined as
youths who, at the time scheduled for testing, were incapacitated, in jail, dead, out of the country, in the
military, or had a language barrier. (See Appendix D for a detailed discussion of out-of-scope youths,
particularly language barrier cases; Tables D-6 and D-8 in Appendix D give full breakouts of out-ofscope youths by type.) About 90.2 percent of youths completed an interview; of those, 78.0 percent also
completed an ASVAB test, and an additional one percent completed the AFQT5 portion of the test.
Further, if we look at response in terms of in-scope (for testing) youths only, nearly 80.6 percent of the inscope youths completed at least the AFQT, and 79.6 percent completed the full ASVAB battery.
Table 5.4 ASVAB Test Status for NLSY97 Youths
Sample
CX
SU
Both Samples
Percent
Percent of Total
of
InterCases
views
Status of Youths
No Interview
Ineligible
Type A2 Nonresponse
Type A1 Nonresponse
Count
Percent
of CX
Cases
100
46
541
1.34
0.62
7.28
45
19
218
1.79
0.75
8.66
145
65
759
1.46
0.65
7.62
Total Non-Interviews
687
9.24
282
11.20
969
9.74
118
1,151
44
63
5,301
1.59
15.48
0.59
0.85
71.30
60
488
30
22
1,710
2.38
19.38
1.19
0.87
67.91
178
1,639
74
85
7,011
1.79
16.47
0.74
0.85
70.44
1.98
18.24
0.82
0.95
78.04
18.60
0.84
0.96
79.59
Total In-Scope for
Testing
6,559
88.22
2,250
89.36
8,809
88.50
98.05
100.00
Total Interviews
6,748
90.76
2,236
88.80
8,984
90.26
100.00
Interview
Out-of-Scope for Testing
Test Nonparticipation
Incomplete AFQT6
AFQT Complete7
Complete Battery
Count
Percent
of SU
Cases
Count
Percent
of InScopes
7,435
100.00 2,518
100.00 9,953
100.00
Total Overall
8
A final summary table, Table 5.5, displays response rates overall and for several breakouts,
including type of area, region, gender, race/ethnicity, and age. Nonmetropolitan youths have consistently
5
AFQT stands for the Armed Forces Qualification Test, calculated as the sum of the number of correct answers to
the Word Knowledge, Paragraph Comprehension, Arithmetic Reasoning, and Mathematics Knowledge subtests.
This represents a portion of the entire ASVAB battery, which consists of twelve subtests in total.
6
An incomplete AFQT is a case for which some but not all of the subtests have been completed; specifically, one
or more of the subtests which make up the AFQT portion of the ASVAB is not complete.
7
An AFQT Complete is a case for which some of the subtests are not complete, but all of the subtests used in the
AFQT are complete.
8
For the purposes of Table 5.5, an AFQT Complete is treated as a response, while an Incomplete AFQT is treated
as a nonresponse.
40
higher response rates than their metropolitan counterparts. Region, which corresponds to the four Census
regions, also shows consistent, though small, differences. While the Midwest has the highest response
rates across the regions, the response rates for all regions are very close and probably do not differ by
more than sampling variability. Differences by gender show females with slightly higher response rates
than males. Race/ethnicity breakdowns show the most disparate response rates; Hispanics respond less
frequently than any other racial/ethnic group. Non-Hispanic, nonblack youths have the highest response
rates, while non-Hispanic blacks fall somewhere in between. Age differences are minimal, though 16year-olds tend to respond at a lower rate than youths of other ages.
Table 5.5 Summary of Response Rates (in Percent), Overall and by Domain
Domain
Household
Screener
94.02
Youth
Interview
Given
Screener
92.19
ASVAB
Given
Interview
78.88
ASVAB
Given
Screener
72.72
Total
Type of Area
MSA*
Non-MSA
93.65
95.67
91.87
93.31
77.72
82.79
71.40
77.25
Region
Northeast
Midwest
South
West
94.87
95.08
93.45
93.21
90.89
94.31
91.72
91.91
78.64
80.75
78.74
77.37
71.48
76.16
72.22
71.11
Gender
Male
Female
92.03
92.37
78.03
79.77
71.81
73.68
Race/Ethnicity
Hispanic
Non-Hispanic Black
Non-Hispanic Nonblack
90.23
92.31
92.95
71.71
77.28
82.53
64.70
71.34
76.71
79.69
79.19
80.85
79.54
76.61
74.11
73.28
74.71
74.16
68.70
Age
12
93.00
13
92.54
14
92.41
15
93.24
16
89.67
* Metropolitan Statistical Area or Consolidated Metropolitan Statistical Area
41
Screener completion rates were also calculated by PSU and segment and are displayed in Figures
5.1 through 5.4 below. As seen in Figure 5.1, screener participation by PSU was higher than 85 percent
for almost all of the PSUs; only eight of the 200 fell below that rate. Figure 5.2 shows the distribution of
nonresponse by PSU, indicating that 90 percent of the PSUs had a nonresponse rate of 10 percent or less.
Figures 5.3 and 5.4 display similar results for segments. While a larger proportion of the segments fall
below the 85 percent completion rate, we can still see in Figure 5.4 that 90 percent of the segments have a
nonresponse rate below 15 percent.
Figure 5.1 Screener Completion
by PSU
Figure 5.2 Distribution of Screener Type A1
Nonresponse by PSU
900
875
850
825
800
775
750
725
700
100%
80%
60%
40%
20%
0%
0%
20%
40%
60%
80%
0%
100%
20%
40%
60%
80%
100%
Nonresponse Rate
Completion Rate
Figure 5.3 Screener Completion
by Segment
Figure 5.4 Distribution of Screener Type A1
Nonresponse by Segment
1800
100%
1500
1200
80%
900
600
40%
300
0
0%
60%
20%
0%
20%
40%
60%
80%
0%
100%
20%
40%
60%
80%
100%
Nonresponse Rate
Completion Rate
Obviously, there were some PSUs and segments that had very low response rates. In fact, four
PSUs and 56 segments fell below a 70 percent screener completion rate, and 11 segments fell below 30
percent. These low rates are due in part to problems with the computers used for screening; in some
segments, computer crashes were the overwhelming cause of screener nonresponse. Appendix N details
the issues regarding crashed cases during the screening process; Appendix M gives a full analysis of
noninterview reports (NIRs) and examines further reasons for screener nonresponse.
Impact of Nonresponse. The impact of nonresponse on the survey results (in this case the
NLSY97 interview responses and the ASVAB test scores) depends on the product of two factors: (1) the
proportion of the sample which is nonresponding, and (2) the magnitude of the difference between
42
respondents and nonrespondents. A large difference between respondents and nonrespondents combined
with a small percentage of nonresponse may produce minimal impact on the survey results. Similarly, a
high rate of nonresponse may have minimal impact on survey results as long as the difference between
respondents and nonrespondents is small. Since the impact of nonresponse is a function of the product of
the two factors, if either factor is small the impact may be minimal.
The major difficulty in estimating the impact of nonresponse derives from the fact that although
the rate of nonresponse (factor 1) is usually calculable, the magnitude of the difference between
respondents and nonrespondents (factor 2) can seldom be known with any confidence. Several methods
for estimating the characteristics of nonrespondents have been suggested in the literature, but since none
is entirely satisfactory, one can assume that the impact of nonresponse is negligible only when the rate of
nonresponse (factor 1) is very small.
One possible way to examine the potential for differences between respondents and
nonrespondents is to examine response rates among subgroups, as shown earlier in Table 5.7. A common
method for reducing the impact of nonresponse is to adjust the weights for nonresponse. For the
NLSY97, corrections for nonresponse were introduced in the weighting process (see Chapter 4). A
correction for screener nonresponse was made at Step 2 and for interview nonresponse at Step 5.
Screener nonresponse was adjusted at the segment level for all respondents in the segment while
interview nonresponse was adjusted separately within design cohorts based upon sample (CX or SU), age,
race/ethnicity, and sex. Thus, in correcting for screener nonresponse we assumed no difference between
responding and nonresponding households in the same segment. In the correction for interview
nonresponse we assumed no difference between respondents and nonrespondents who were in the same
design cohort.
As described in Chapter 4, a further adjustment to correct for differences between study
participants and the appropriate reference population (the CPS was used) was made in the calculation of a
post-stratification adjustment factor. The use of age, sex, and race/ethnicity in the post-stratification
adjustment was based on theoretical considerations as well as previous survey experience which suggests
that these factors are related to many survey variables of interest.
It is important to point out that all surveys with less than 100 percent response are forced to make
assumptions about the characteristics of nonrespondents. An unweighted (or self-weighting) survey
assumes that all nonrespondents taken as a group are like all respondents taken as a group. We found it
preferable on both theoretical and empirical grounds to make such assumptions at a disaggregated level,
i.e., at the segment or design cohort level. We feel that these more exacting corrections treat nonresponse
with the care it deserves.9
Having made the various adjustments for nonresponse described above, we carried out a simple
but fairly standard test to examine differences between the final completed sample and the sample at
various stages in the overall survey process, in addition to differences between the NLSY97 results and a
nationally representative reference population. Specifically, we compared the sample distribution for a
few variables for screener respondents, NLSY97 eligibles who completed the interview, and NLSY97
9
Needless to say, we also did all we could through field operations to keep the response rate high.
43
respondents who completed the ASVAB test to families with resident youths aged 12 to 16 responding to
the March 1997 Current Population Survey (CPS). Table 5.6 shows the results of these comparisons for
the NLSY97 sample. The rationale for making the comparisons is to see whether there are any
differences between the CPS sample of youths and the NLSY97 sample, and whether any significant
changes in sample composition appear from one stage of the survey to the next as nonresponse attrition
reduces the number of participants. Distributions are also given separately for the CX and the SU
samples; this will help to illustrate the differences between these two distinct groups and illuminate
where, in particular, any differences between NLSY97 and CPS may occur. As Table 5.8 shows, the
overall sample of screener respondents is almost identical to the overall sample of NLSY97 interviews
and test completes in terms of the distribution across household size, urbanicity, and region of the
country.10 Further, NLSY97 displays a somewhat higher proportion than CPS of youths in small
households and in metropolitan areas. Most of the differences by region are due to the concentration of
supplemental cases in the South; however, even the CX sample has more youths from the South and
fewer from the West than the CPS. Remember that the SU sample consists only of black and Hispanic
youths and was sampled with probability proportional to the number of blacks and Hispanics; thus, this
sample is very different from the CPS and the CX samples, which are balanced, representative cross
sections of the U.S. population. Because the SU sample distribution is skewed toward black and Hispanic
households, the numbers for this sample will be quite different from the other samples shown. For this
reason, the reader should focus attention on a comparison of the CPS sample to the CX sample.
10
Since sample weighting was only applied to respondents who participated in the interview, the comparison
between the screener sample and the interview sample is shown on an unweighted basis.
44
Table 5.6 Distributions (Unweighted) of Screener Variables for CPS Youths, Completed Screener Respondents,
and NLSY97 Interview and Test Completes11
Distribution (in percent) among
Variable
CPS
Youths
Aged 12-16
Screener Respondents
(NLSY97 Eligibles)
CX
SU
Total
NLSY97 Interviews
CX
SU
Total
NLSY97 Test Completes
CX
SU
Total
Persons in Household
1–2
3
4
5
6 or more
4.8
17.8
33.4
24.1
19.9
5.6
18.4
33.6
25.1
17.3
7.1
16.5
24.9
20.4
31.1
5.9
18.0
31.4
23.9
20.8
5.4
18.7
33.5
25.4
17.0
6.5
15.7
25.2
20.4
32.2
5.6
18.0
31.4
24.2
20.8
4.9
18.6
34.2
25.4
16.9
6.4
15.5
26.4
20.3
31.4
5.2
17.9
32.3
24.2
20.4
Type of Area
MSA
Non-MSA
74.1
25.7
74.2
25.8
87.1
12.9
77.5
22.5
73.9
26.1
86.8
13.2
77.2
22.8
72.9
27.1
85.6
14.4
76.0
24.0
Region
Northeast
Midwest
South
West
18.7
23.4
29.8
28.1
19.0
25.8
33.9
21.3
15.3
12.2
47.1
25.3
18.0
22.3
37.3
22.4
18.7
26.3
33.9
21.1
14.6
13.0
47.3
25.1
17.7
22.9
37.2
22.1
18.1
26.6
34.0
21.1
15.8
12.5
48.0
23.7
17.5
23.3
37.3
21.9
11
These percentages are slightly inaccurate due to ineligibles found in Round 2 and subsequent reweighting for Round 1.
45
Table 5.6 shows that nonresponse did not produce any major changes in sample composition at
later stages of the survey process, at least for the variables presented here. Yet the limited number of
screener variables available for comparison should induce caution in concluding that the distributions of
test scores and study variables among nonrespondents would be identical to the distributions among
respondents.
More importantly, these analyses show that the NLSY97 sample is not dramatically different
from another nationally representative sample, the CPS. Further evidence of the similarities between the
NLSY97 and the CPS based on family income and educational attainment of the youths’ parents can be
found in Appendices F and G.
In conclusion, we would like to point out that the best overall evidence that the impact of
nonresponse must be small is the relatively high response rate attained. Whatever differences there may
be between respondents and nonrespondents are diluted by the fact that there are many more respondents
than nonrespondents. For example, imagine that we wanted to consider the equation
p = rp1 + (1 − r ) p2
where p is the true proportion of youths with a certain attribute, p1 is the proportion for participants, p2 is
the proportion for nonparticipants, and r is the response rate. Normally, because we don’t have any
information about nonparticipants, we employ an unbiased estimator of p1 . As Table 5.7 shows, for p =
.5 and for response rates ranging from 72 to 82 percent, even if p2 is as much as 10 percentage points
higher than p1 , the true proportion is still quite close to p1 . In other words, a large difference between
respondents and nonrespondents still produces only a small difference in the true versus observed
proportion, given that the response rate is relatively high.
Table 5.7 True Overall Proportions of Youths Given Varying Proportions for
Nonparticipants and Varying Response Rates
p1 : proportion for
participants
0.50
0.50
0.50
0.50
0.50
0.50
p2: proportion for
nonparticipants
0.55
0.55
0.55
0.60
0.60
0.60
r: response rate
0.82
0.77
0.72
0.82
0.77
0.72
p: true overall
proportion
0.509
0.512
0.514
0.518
0.523
0.528
5.2 Refielding
While all of the response rates presented in Section 5.1 are final, the rates of response achieved
by the end of the field period in October of 1997 were actually somewhat lower. After completion of the
field period, NORC discovered that many of the screeners that were not completed had, in fact, been
46
broken off in midstream or otherwise left unfinished; furthermore, there was evidence that these
incomplete cases were rich in NLSY97-eligible youths. In an effort to capture at least some of these
youths, NORC went back into the field in what we refer to as the “refielding” period to complete the
screening of these households and to complete the youth and parent interviews for NLSY97-eligible
youths found therein. We refielded 1,175 such households in April-May 1998.
Also, because we were going back into the field, we decided to reapproach 640 youth cases in
527 households who were screened in during the original field period, but for whom the youth interview
had been refused or was otherwise incomplete. During refielding, we sought to obtain the youth and
parent interviews for these cases. Overall, we refielded 1,702 (1,175 + 527) households.
Table 5.8 displays screener response statuses before and after refielding; the corresponding rates
by sample type (CX and SU) are shown in Tables 5.9 and 5.10. Of the 1,702 households refielded, 973
completed a screener, compared to 763 completed screeners before refielding. This corresponds to 210
additional completed screeners, 161 (697-536) of which were achieved in the CX sample and 49 (276227) in the SU sample.
Table 5.8 Screener Response Status Before and After Refielding
Final Screener Response Status
Response Status
Before Refielding
Type C Non-Interview
Type B Non-Interview
Type A2 Non-Response
Type A1 Non-Response
Completed Screener
TOTAL
Type C
NonInterview
4
0
0
1
3
8
Type B
NonInterview
1
7
0
3
1
12
Type A2
NonResponse
0
0
13
0
2
15
Type A1
Non- Completed
Response Screener
6
9
21
7
21
27
556
263
90
667
694
973
TOTAL
20
35
61
823
763
1,702
Table 5.9 Screener Response Status Before and After Refielding: CX Sample
Final Screener Response Status
Response Status
Before Refielding
Type C Non-Interview
Type B Non-Interview
Type A2 Non-Response
Type A1 Non-Response
Completed Screener
TOTAL
Type C
NonInterview
2
0
0
0
1
3
Type B
NonInterview
1
5
0
3
0
9
47
Type A2
NonResponse
0
0
11
0
1
12
Type A1
Non- Completed
Response Screener
6
6
13
5
14
17
450
201
66
468
549
697
TOTAL
15
23
42
654
536
1,270
Table 5.10 Screener Response Status Before and After Refielding: SU Sample
Final Screener Response Status
Type C
Response Status
NonBefore Refielding
Interview
Type C Non-Interview
2
Type B Non-Interview
0
Type A2 Non-Response
0
Type A1 Non-Response
1
Completed Screener
2
TOTAL
5
Type B
NonInterview
0
2
0
0
1
3
Type A2
NonResponse
0
0
2
0
1
3
Type A1
Non- Completed
Response Screener
0
3
8
2
7
10
106
62
24
199
145
276
TOTAL
5
12
19
169
227
432
Table 5.11 gives NLSY97 response status before and after refielding, illustrating the NLSY97eligible cases screened in and interviewed in the refielding operation. As seen here, 278 new NLSY97
youth cases were spawned during refielding, 2 of which were ultimately found to be ineligible, and 222 of
which completed an interview. Further, of the 640 NLSY97 cases coded as type-A1 nonrespondents
before refielding, 12 were found to be ineligible, and 180 actually completed an interview during
refielding. Thus, the refielding operation resulted in an additional 402 completed interviews.
5.11 Response Status for the Youth Interview Before and After Refielding
Response Status
Before Refielding
Screened in Refielding
Type A1 Non-Response
Final Response Status
Type A2 Non- Type A1 NonResponse
Response
Ineligible
2
12
14
1
2
3
53
446
499
Completed
Interview
222
180
402
TOTAL
278
640
918
The refielded cases were clearly hard to interview, either because they initially refused the youth
interview or because the original screener was broken off or incomplete. Yet, we ultimately found 918
youths in the 1,702 refielded households, or a youth rate of 0.54 youths per household. This youth rate is
much larger than the rate for American households overall. The refielded households were clearly rich in
eligible youths. And since this is so, the refielding experience begs the question, “Were other hard to
enumerate households, not refielded, richer than average in youths?” Modern professional experience
and judgement suggests that the answer to this question must be “yes,” yet at this writing we have no hard
data to confirm or refute this judgement. We conclude, based only on judgement, that some portion of the
shortfall between youths planned and youths achieved may be due to hard to enumerate households, rich
in youths, who did not complete the NLSY97 screener.
48
5.3
Representativeness of the Sample
Overview. In this section, we examine the representativeness of the NLSY97 sample, including
the large screening sample and the youths who actually participated in the NLSY97 interview and testing
program. We begin by reviewing the screening sample in terms of the coverage of housing units and the
designation of housing units as occupied or vacant. This review is intended to shed light on the quality of
the sampling of PSUs and segments, the listing of housing units or addresses, the sampling of housing
units or addresses for screening, and the screening for occupancy status.
Next, we examine the screening sample in terms of the coverage of persons, focusing on the 1223 age range. We present the age distribution of all persons who were screened in our large screening
sample, and we discuss the distribution of screened households by the number of eligible youths reported.
Third, we analyze the age/sex/race distribution of the youths who screened in and of the youths
who participated in the NLSY97. This analysis is intended to provide an understanding of the
demographic balance of the samples and of how the sample compares to known or presumed known
national norms. We also review the composition of the sample with respect to its socio-economic
balance, including a review of variables such as parent education and household income.
Fourth, we examine once again the coverage of youths by age, sex, and race, and we compare and
contrast the coverage of NLSY97, the Current Population Survey (CPS), and the Decennial Census.
By this point, it will have become clear to even the most casual reader that NLSY97 experienced
some notable undercoverage of youths aged 12-23. Finally, we summarize some special studies we
performed to try to explain why the undercoverage occurred. Such studies include validation interviews
and debriefing interviews.
Coverage of HUs. For every segment selected, we have the 1990 Census count of housing units
as well as counts of youths under the age of 18 in each of three race/ethnicity categories: Hispanic, nonHispanic black, and non-Hispanic nonblack. We also have, for each segment, a sampling base weight,
which is the inverse of the segment’s probability of selection.
The sampling base weight indicates the number of units “represented” by each unit in the
segment. Therefore, if a segment has five Hispanic youths under the age of 18 years and a sampling base
weight of 1000, that segment is said to “represent” (5 * 1000 = ) 5000 Hispanic youths under the age of
18. A “representative” sample would therefore “represent” the entire population of the United States. We
refer to this process as “weighting up the segment-level information.”
We analyze the CX and SU samples separately because they were selected independently. Table
5.12 below shows the outcome of this analysis for the CX sample for all four counts mentioned above.
The analysis is done separately for each of the four Census regions in addition to the national analysis.
Four rows of data are given for each analysis. The first line is the data from our selected segments
weighted up by the segment base weight. The second line is the 1990 Census count of youths aged 0 to
17 or housing units. The third line is the difference, and the fourth line is this difference as a percentage
of the data from the 1990 Census.
49
Table 5.12 Weighted-up 1990 Census Counts of Youths Under Age 18
and of Housing Units for the CX Segments
Total Youths
Hispanic
Non-Hispanic
Black
All Other
Housing Units
National
“Represented”
Census
Difference
Percent Difference
62,210,920
63,606,544
-1,395,624
-2.19
7,071,404
7,637,394
-565,990
-7.41
9,772,687
9,566,954
205,733
2.15
45,366,828
46,402,196
-1,035,368
-2.23
102,184,234
102,263,678
-79,444
-0.08
Northeast
“Represented”
Census
Difference
Percent Difference
11,889,060
11,911,083
-22,023
-0.18
1,059,730
1,164,579
-104,849
-9.00
1,726,455
1,652,022
74,433
4.51
9,102,875
9,094,482
8,393
0.09
21,232,281
20,810,637
421,644
2.03
Midwest
“Represented”
Census
Difference
Percent Difference
15,267,945
15,630,137
-362,192
-2.32
574,694
634,725
-60,031
-9.46
1,704,364
1,883,154
-178,790
-9.49
12,988,887
13,112,258
-123,371
-0.94
24,304,554
24,492,718
-188,164
-0.77
South
“Represented”
Census
Difference
Percent Difference
21,130,305
22,017,465
-887,160
-4.03
1,722,438
2,247,486
-525,048
-23.36
5,408,463
5,167,288
241,175
4.67
13,999,403
14,602,691
-603,288
-4.13
35,692,925
36,065,102
-372,177
-1.03
West
“Represented”
Census
Difference
Percent Difference
13,923,610
14,047,859
-124,249
-0.88
3,714,542
3,590,604
123,938
3.45
933,405
864,490
68,915
7.97
9,275,663
9,592,765
-317,102
-3.31
20,954,475
20,895,221
59,254
0.28
Region
50
Table 5.13 below shows a similar analysis for the SU sample. However, because the SU sample
was intended to be representative only for black and Hispanic youths, we do not present this analysis for
the housing units or the non-Hispanic, nonblack youths in the SU sample.
Table 5.13 Weighted-up 1990 Census Counts of Minority Youths
Under Age 18 for the SU Segments
Total Minority
Youths
Hispanic
Non-Hispanic
Black
“Represented”
Census
Difference
Percent Difference
17,067,764
17,204,348
-136,584
-0.79
7,521,903
7,637,394
-115,491
-1.51
9,545,861
9,566,954
-21,093
-0.22
Northeast
“Represented”
Census
Difference
Percent Difference
2,733,764
2,816,601
-82,837
-2.94
1,113,334
1,164,579
-51,245
-4.40
1,620,430
1,652,022
-31,592
-1.91
Midwest
“Represented”
Census
Difference
Percent Difference
2,505,257
2,517,879
-12,622
-0.50
726,056
634,725
91,331
14.39
1,779,202
1,883,154
-103,952
-5.52
South
“Represented”
Census
Difference
Percent Difference
7,489,582
7,414,774
74,808
1.01
2,182,927
2,247,486
-64,559
-2.87
5,306,655
5,167,288
139,367
2.70
West
“Represented”
Census
Difference
Percent Difference
4,339,161
4,455,094
-115,933
-2.60
3,499,587
3,590,604
-91,017
-2.53
839,574
864,490
-24,916
-2.88
Region
National
The results for youths are quite encouraging, especially for the SU sample. The number of black
youths under the age of 18 is only off nationally by 0.2 percent. Even at the regional levels (for which the
sample was not designed to be representative), none of the differences for black youths are over 6 percent.
The national number of Hispanic youths under the age of 18 is only off by 1.5 percent. The discrepancy
of 14.4 percent in the Midwest looks large, but the absolute differential of 91,331 is not much larger than
the discrepancy of 91,017 in the West. It is not surprising that the SU sample shows smaller percentage
discrepancies than the CX sample. The targeting of the SU sample to black and Hispanic youths results
51
in more accuracy for these two groups at the expense of accuracy for non-minority youths and housing
units.
The CX sample is extremely close with respect to housing units, coming within 0.08 percent of
the Census count. In fact, this difference would be zero except that one segment was selected with
certainty. The CX sample is very representative with respect to black and non-minority youths under the
age of 18. There are 2.2 percent more black and 2.2 percent fewer non-Hispanic, nonblack youths
represented by the sample than appear in the 1990 Census. There is a shortfall among Hispanic youths,
however. There are 7.4 percent fewer Hispanic youths under the age of 18 represented by the sample
than appear in the 1990 Census. This difference is almost entirely due to a large shortfall in the South
region. It is important to remember, though, that the CX sample is not targeted at minorities; a segment’s
selection probability is proportional to its total number of housing units. In a nationally representative
sample, there is often variability in the size of subgroups.
When NORC sent listers to the segments selected in the Fall of 1995, there were usually not the
same number of housing units in the segment as the 1990 Census indicated. Many segments had
undergone significant change between 1990 and 1995. In order to show the representativeness of the
housing units that were actually found in the selected segments, we can weight up the number of housing
units listed in the segments (again using the segment base weight, the reciprocal of the segment
probability of selection). The number of housing units thus Arepresented@ is shown below, compared to
the 1990 Census as well as more recent Census Bureau estimates. This analysis tests the quality of the
NLSY97 listing operation. The results for the SU sample are more difficult to interpret because of the
measure of size used in selecting PSUs and segments.
Table 5.14 below shows weighted-up counts of HUs for the CX sample.
Table 5.14 Weighted-Up Listing Counts for the CX Sample
(3)
Sample
Listing
(3)/(2)
Ratio
(in
percent)
(4)
Population
Estimates
Program
July, 1996
(3)/(4)
Ratio
(in
percent)
Region
(1)
1990
Census
(2)
Sample
Census
(2)/(1)
Ratio
(in
percent)
TOTAL
102,263,678
102,184,234
99.92
106,944,539
104.66
109,800,000
97.40
Northeast
20,810,637
21,232,281
102.03
21,672,550
102.07
21,530,000
100.66
Midwest
24,492,718
24,304,554
99.23
25,189,511
103.64
26,014,000
96.83
South
36,065,102
35,692,925
98.97
38,441,114
107.70
39,416,000
97.53
West
20,895,221
20,954,475
100.28
21,641,365
103.28
22,840,000
94.75
52
Column (1) contains 1990 Census housing unit counts, while column (2) contains the weightedup 1990 Census housing unit counts from the selected NLSY97 segments (these numbers also appear in
Table 5.12). The third column shows that columns (1) and (2) are within one percent overall, and within
two percent for every region. Column (4) shows (revised) estimates of the numbers of housing units as
of July 1, 1996, from the Census Bureau’s Population Estimates Program (PEP) 12. The PEP estimates the
number of housing units by updating the 1990 Census counts of housing units. The main sources for the
updates are construction and demolition permits, where these permits are available. These estimates are
larger than those in column (1), indicating that there are more housing units now than in 1990.
Column (3) shows the weighted-up sums of actual housing unit counts from the NLSY97 listing
operation. The weighted-up sums of housing unit counts from the NLSY97 listing effort are greater than
the 1990 Census counts, but are smaller than the July 1996 estimates. This is to be expected because the
listing operation was done in the Fall of 1995.
Under the assumption that growth in the housing stock was uniform between 1990 and 1996, we
might expect the numbers in column (3) to be very close to the July 1996 PEP estimates. In fact, the
numbers in column (3) are closer to the July 1996 PEP estimates than to the 1990 Census counts, and
there are a few factors that could explain why they are not even closer to the July 1996 PEP estimates (we
would not expect our listing-based estimate to be equal because of a half-year of growth between the Fall
of 1995 and July 1996). First, the PEP estimates themselves are estimates, and therefore subject to error.
Second, the numbers in column (3) are based on a sample, and are therefore subject to sampling
variability. Finally, there is a possibility of NLSY97 or PEP coverage error. We note, however, that
NLSY97 used a half-open interval procedure (see Chapter 3) during the main field effort to correct any
such listing errors as well as to include new construction in our sample.
Some of the comparisons made above for the CX sample are inappropriate for the SU sample
because the probabilities of selection were based upon minority youths rather than housing units. This
means that the weight (the inverse of the probability of selection) will be large for a non-minority
segment and small for a minority segment. Therefore, non-minority segments will have their housing unit
counts multiplied by a large weight, regardless of how many housing units they contain. Any nonminority segments with a large number of housing units will dominate the sum, which results in an
unbiased estimator of the 1990 Census count of housing units, but with a very large sampling variance.
One extra or fewer non-minority segment will have a large effect on the total sum of weights. However,
it is still useful to compare the weighted-up listing operation counts with the weighted-up 1990 Census
segment counts. This comparison is shown in Table 5.15.
12
The Census estimates given in Table 5.14 can be found at
http://www.census.gov/population/estimates/housing/prhuhht1.txt. More information can also be found at this
website, and the methodology used is described in Census report P25-1127.
53
Table 5.15 Comparison of Census and Listing Counts in the SU Sample
(2)
Sample
Census
(3)
Sample
Listing
(3)/(2)
Ratio
(in percent)
115,405,337
118,265,606
102.4785
Northeast
13,808,939
15,745,940
114.0272
Midwest
54,819,502
52,826,805
96.3650
South
32,416,072
33,438,963
103.1555
West
14,360,825
16,253,898
113.1822
TOTAL
The weighted-up Census housing unit counts for the SU sample shown as column (2) of Table
5.15 can be used in order to obtain a test of the listing operation for the SU sample. This is because
weighting-up the actual listing counts to a national total uses the same weights as those used in column
(2). The weighted-up actual listing counts are shown in column (3) of Table 5.15. The rightmost column
in Table 5.15 shows that these two columns are within 3 percent overall, and within 2,000,000 housing
units in every region. These differences do seem to show a general growth from the 1990 Census
[column (2)] to the 1995 listing operation [column (3)], which was expected. It is also true that
differences found in segments with large weights would be exaggerated by the large weights. This could
explain the apparent decrease in the Midwest, which is where the large weights (i.e., areas with low
counts of minority youths) seem to be concentrated.
Of the 270,197 housing units listed, 90,139 were selected into the sample; a further 818 housing
units were added to the sample through the half-open interval procedure. We can weight up these 90,957
housing units using their (untruncated) base weights. This is done in Table 5.16 below.
Table 5.16 Sums of Base Weights for Selected Housing Units
Sample
CX
SU
Region
TOTAL
Northeast
Midwest
South
West
TOTAL
Northeast
Midwest
South
West
Sum of Weights
108,965,394
22,070,137
24,579,151
39,489,719
21,926,387
121,182,555
15,761,767
55,976,275
33,140,334
16,304,180
These numbers are very similar to the numbers in column (3) of Tables 5.14 and 5.15, indicating that a
54
representative sample of the listed housing units was selected. It has already been stated that the sum of
the weights for the SU sample is too high because of the large weights assigned to housing units in nonminority areas, and that the sum of the weights for the CX sample is close to the national total number of
housing units.
We now move from an examination of the sampling of PSUs, segments, and HUs to an
examination of the results from screening, where field interviewers classified HUs as occupied or vacant.
Prior to the field work, it was hypothesized that 85 percent of the housing units selected would be
occupied. We can examine the actual rate because of the very complete set of screener disposition codes
used for NLSY97. Table 5.17 shows the NLSY97 occupied rate, and compares it to two Census Bureau
standards. One standard used is the 1996 Housing Vacancy Survey. This survey comes from Current
Population Survey household estimates averaged over a twelve-month period. The second standard is the
Population Estimation Program used already in this chapter. NLSY97 has a very similar rate of vacant
housing units (11.3%) to the rate shown in the Census Bureau’s 1996 Housing Vacancy Survey (11.5%),
and is slightly larger than the rate shown in the Census Bureau’s (revised) July 1996 Population Estimates
Program (10%).
Table 5.17 Comparison of the NLSY97 Vacancy Status of Housing Units
All Lines
Not an HU
* Business
* Group Quarters
NLSY97/PAY97
Cases Percent
90,957
Housing Vacancy
Survey*
Cases Percent
Population Estimates
Program#
Cases Percent
327
233
94
HU
90,630
Vacancy not determined
157
* HU inaccessible
157
Vacancy determined
90,473
100.00 114,139,000
100.00 109,800,000
100.00
++Vacant
10,214
11.29
13,155,000
11.53
11,049,000
10.06
* Condemned
163
0.18
* Demolished
163
0.18
* No such address, HU
1,580
1.75
* Vacation cabin
600
0.66
* Unusable as perm. Resid.
121
0.13
* Transient use
102
0.11
* Under construction
28
0.03
* Vacant
7,457
8.24
++Occupied
80,259
88.71 100,984,000
88.47
98,751,000
89.94
*
These data are available at http:www.census.gov/hhes/www/housing/hvs/historic/histab7.html
#
From the (revised) Population Estimates Program: July, 1996.
As stated in Chapter 4, the preliminary coverage ratios that suggest that “one out of every three
NLSY97 eligibles” were missed are very misleading. The caps used during weighting steps 1 and 2 to
55
prevent extreme weights have a negative effect on the coverage ratios. The effect of these weighting caps
can be seen most effectively from the effects on the sum of the household weights, examined in Table
5.18 below.
Table 5.18 Effect of the Weighting Caps on the Sums of the Household Weights
Census estimate of housing units, 1996 (PEP)
NLSY97 estimate of housing units
Census estimate of occupied housing units, 1996 (PEP)
NLSY97 estimate of occupied housing units
Percent of difference
Sum of weights (base weight only capped)
Percent of weights truncated by base weight cap
Sum (base weight and missed HU adjustment capped)
Percent of weights truncated by missed HU adjustment
Sum of weights after nonresponse (no cap of 2.0 used)
Percent of weights truncated by using PSU response rate
Sum of weights (with truncation within 4 PSUs)
Percent of base weights truncated within 4 PSUs
Total reduction by caps in sum of weights
Total percent of base weights truncated by caps
CX Sample
109,800,000
108,965,394
SU Sample
109,800,000
121,182,555
Sum of
Samples
219,600,000
230,147,950
98,751,000
96,413,677
2.37
96,413,677
0.00
95,696,559
0.73
95,092,819
0.61
95,092,819
0.00
1,320,858
1.34
98,751,000
109,531,766
-10.92
83,471,399
26.39
83,374,766
0.10
83,048,494
0.33
82,824,571
0.23
26,707,195
27.04
197,502,000
205,945,443
-4.28
179,885,076
13.19
179,071,325
0.41
178,141,313
0.47
177,917,390
0.11
28,028,053
14.19
The third column in Table 5.18 is simply the sum of the CX and SU weight sums; it is used to
examine an “overall” weighting cap effect. The top row in the table is the Census Bureau’s estimate of the
total number of U.S. housing units, as taken from the 1996 Population Estimates Program (PEP). The
NLSY97 estimate is the total sum of the untruncated base weights after the untruncated missed housing
unit adjustment. The next two rows remove housing units from the rest of the table. The Census estimate
of occupied housing units is also taken from the 1996 PEP. The NLSY97 estimate is the total sum of the
untruncated base weights after the untruncated missed housing unit adjustment for only occupied housing
units. The denominator for all percentages is the PEP estimate of occupied housing units. Base weights
were capped only in the SU sample, but this resulted in a 26.4 percent reduction in the sum of the
supplemental weights.
Other truncations made were: 1) truncating the missed housing unit adjustment to 5.0; 2) (not
technically a truncation) using the PSU response rate instead of the segment response rate in segments
with fewer than 25 respondents to reduce the sum of weights by preventing some large adjustments; and
3) truncating the nonresponse weighting adjustment within the four supplemental PSUs with a low PSU
response rate. These other truncations had only small effects on the sum of the weights (all of less than
one percent).
Taken together, weighting caps decreased the sum of the CX weights by 1.3 percent, the sum of
the supplemental weights by 27.0 percent. Of course, the reader will recall that the weight is restored by
poststratification, the last step in weighting.
56
Coverage of Persons. Chapter 3 touches on the shortfalls in the numbers of youths screened in
to the NLSY97 samples. This section furthers that discussion. We have conducted various analyses to
determine both the coverage and representativeness of the NLSY97 samples. Some of those analyses
were presented in Section 5.1; several more will be illustrated here. The March 1997 CPS was used in all
of the following analyses as a comparison; weighted CPS numbers include the Hispanic supplement,
while unweighted analyses using CPS data do not.
Figures 5.5, 5.6, and 5.7 display weighted13 counts of persons for the overall screening sample as
well as the CX and SU samples compared to the CPS. These figures show distributions for ages 0 to 35
and illustrate the shortfalls in the age ranges critical to the NLSY97. While screening identified most of
the expected persons below age 12, and almost exactly the number of expected persons aged 24 to 35,
there is a large undercount of youths aged 12 to 23.
The reader will recall that youths aged 12-16 were eligible for the NLSY97 and that older youths
up to and including age 23 were eligible for components of PAY97, the DoD study that employed the
NLSY97 screening sample. Thus, the hole in the age distribution coincides exactly with the range of ages
for which the screening sample was intended. No natural age distribution exhibits this type of pattern.
Evidently, the NLSY97 screening sample experienced an undercoverage mechanism specific to its
targeted ages.
Figure 5.5 W eighted C ounts of Persons,
Total Sam ple
5,000,000
4,000,000
3,000,000
2,000,000
1,000,000
0
CPS-Expected
NLSY
0
4
8 12 16 20 24 28 32
Age
13
This weighted analysis relies on a special set of household weights for the screener completed cases. These
weights are different from the final youth weights calculated for youth completes. The special weights, described in
detail in Appendix S, make appropriate allowance for the fact that proxy and gatekeeper screeners did not result in
HH rosters.
57
Similar patterns exist by sample type, as seen below. The CX and SU samples both have
noticeable gaps in coverage for youths aged 12 to 23; for the SU sample, these gaps are even more severe
in the 20- to 23-year-old age range. For all three figures, notice the spike in the number of persons aged
11, and the marked drop following it for youths aged 12. This pattern could indicate that the respondents
were somehow aware of the ages sought for the survey, and thus reported youths who were actually 12
(or older) as 11 years old. Similar age misreporting may also have happened at the other end; the low
number of 23-year-olds reported followed by a sharp spike at age 24 supports this hypothesis.
Figure 5.7 Weighted Counts of Persons,
SU Sample
Figure 5.6 Weighted Counts of Persons,
CX Sample
5,000,000
4,000,000
3,000,000
2,000,000
1,000,000
0
5,000,000
4,000,000
3,000,000
2,000,000
1,000,000
0
CPS-Expected
NLSY
CPS-Expected
NLSY
0 5 10 15 20 25 30 35
0 5 10 15 20 25 30 35
Age
Age
As these figures show, undercoverage is differential by age—specifically by the eligible ages—
therefore, the undercoverage mechanism cannot be one that generally misses households or people. In
other words, the mechanism is not related to sampling but is likely related to reporting in the screening
interview.
Table 5.19 displays the distribution of households by the number of 12- to 16-year-old youths for
the CPS compared to the NLSY97 sample. In this and the tables that follow, please note that a
comparison of the CX sample to the CPS is appropriate because the SU sample is dominated by black and
Hispanic households. As seen here, the NLSY97 sample has a larger proportion of households with no
youths in the age range, further suggesting that households did not report youths when there were in fact
resident youths in the age range.
Table 5.19 Distribution of Households by Reported Number of Youths Aged 12 to 16
Number of
Youths
CPS
Weighted
0
1
2
3
4
5+
.848
.114
.034
.004
.000
.000
Proportion of Households
NLSY97-Weighted
Total
CX
SU
.892
.079
.026
.004
.000
.000
.876
.091
.029
.003
.000
.000
58
.894
.077
.026
.004
.000
.000
Table 5.20 gives conditional household distributions, given there was at least one resident youth
in the 12- to 16-year-old age range, this time overall and by race/ethnicity of the householder. After
controlling for the existence of at least one youth, the overall NLSY97 sample has slightly fewer
households with only one such youth, and slightly more with two or more such youths. Again, this
implies that households with only one youth in the age range were less likely to report that youth, while
households with multiple youths were more likely to report all of their youths. This finding holds for the
sample overall and for each of the race/ethnicity groups.
Table 5.20 Conditional Distribution of Households (HHs) by Reported Number of Youths Aged
12 to 16, Given at Least One Such Youth
Number of
Youths
CPS
Weighted
Proportion of Households
NLSY97-Weighted
Total
CX
1
2
3
4
5+
Total HHs
.748
.224
.024
.003
.001
.725
.238
.033
.003
.001
.722
.240
.034
.003
.001
.733
.235
.028
.003
.000
1
2
3
4
5+
Hispanic
Householder
.733
.237
.025
.006
.000
.675
.276
.045
.003
.000
.673
.280
.044
.002
.000
.694
.232
.066
.007
.000
1
2
3
4
5+
Non-Hispanic,
Black Householder
.750
.214
.026
.006
.004
.738
.211
.041
.007
.003
.739
.210
.041
.006
.003
.737
.208
.039
.015
.001
1
2
3
4
5+
Non-Hispanic,
Nonblack
Householder
.754
.221
.023
.002
.001
.731
.238
.029
.002
.000
.728
.239
.030
.002
.000
.737
.240
.022
.001
.000
SU
The next table, Table 5.21, displays the mean number of youths in each household that has at
59
least one youth aged 12 to 16. Again, these tables show that the NLSY97 sample has larger numbers of
youths in the 12- to 16-year-old age range given that they reported at least one youth in that age range,
once again suggesting that households with more than one youth were more likely to report their youths
than households with only one youth. Again, this is true overall and for each of the races/ethnicities.
Table 5.21 Conditional Mean Youths, Age 12 to 16, Per Household
Mean Number of Youths, Age 12-16,
Per Household With >= 1 Such Youth
NLSY97-Weighted
CPS Weighted
Total
CX
Race/Ethnicity
SU
Hispanic
1.303
1.377
1.376
1.386
Non-Hispanic, Black
1.300
1.328
1.324
1.337
All Other
1.275
1.303
1.308
1.288
Total
1.282
1.316
1.320
1.302
A tabulation of average household sizes is shown in Table 5.22, and the distribution of
households by the size of the household is given in Table 5.23. Compared to the CPS, NLSY97
households are reported smaller. Table 5.22 shows that the average household size for the March, 1993
CPS, which was used in planning the NLSY97 sample size, is 2.62; this falls between the mean household
size for the March, 1997 CPS (2.89) and for the NLSY97 (2.34). Also, Table 5.23 indicates that more
NLSY97 households report one or two residents, and fewer households report three or more residents,
than CPS. This is in accordance with the evidence presented earlier, suggesting that households with
small numbers of youths are less likely to report them.
Table 5.22 Average Household Size
Sample
Number of Persons
CPS, March 1993– Weighted
CPS, March 1997 – Weighted
NLSY97 – Weighted
2.62
2.89
2.34
NLSY97 – Weighted (CX PSUs)
NLSY97 – Weighted (SU PSUs)
2.34
2.41
60
Table 5.23 Distribution of Households by Size of Household
Number of
Residents
NLSY97/PAY97-Weighted
Total
CX
CPS Weighted
.252
.324
.169
.152
.067
.036
1
2
3
4
5
6+
.317
.339
.146
.123
.051
.024
.318
.340
.145
.121
.051
.024
SU
.290
.340
.158
.135
.056
.022
Table 5.24 shows that the NLSY97 screening sample captured fewer youths between the ages of
12 and 23 than we expected to exist in the population at large. The CPS numbers indicate that youths
between ages 12 and 23 make up between 16.3 and 16.7 percent of total people, while for the NLSY97
sample they make up only 12.7 percent. Again, the results of the 1993 CPS fall somewhere between
those of the 1997 CPS and the NLSY97/PAY97, except for 18- to 23-year-olds.
Table 5.24 Youths as a Proportion of Total Persons Screened
Age Range of
Youths
March 1993
CPS-Weighted
March 1997
CPS-Weighted
NLSY97/PAY97-Weighted
Total
CX
12 to 16
.069
.073
.061
.060
.067
18 to 23
.081
.079
.055
.055
.050
12 to 23
.163
.167
.127
.126
.128
SU
Examination of Demographic Balance. Up to this point, we have presented figures and tables
detailing the shortfalls in the NLSY97 counts of youths. Tables 5.25, 5.26, and 5.27 illustrate the
population proportions in various demographic domains compared to the CPS. The tables demonstrate
that, though there are obvious shortfalls in overall counts, there is little evidence that the sample is not
demographically balanced.
61
Table 5.25 Race by Gender Percentages for CPS vs. NLSY97-Eligible Sample14
Race/
Ethnicity
CPS Ages 12-16
Gender
Total
Male Female
53.7
46.3
100.0
10.6
9.8
10.2
NLSY97 Total Eligible
Gender
Total
Male
Female
51.5
48.5
100.0
21.8
21.6
21.7
NLSY97 CX Eligible
Gender
Total
Male
Female
51.5
48.5
100.0
14.0
14.0
14.0
NLSY97 SU Eligible
Gender
Total
Male Female
51.4
48.6
100.0
45.0
43.7
44.4
Hispanic
% w/in race
% w/in sex
Non-Hispanic
Black
% w/in race
% w/in sex
51.2
12.5
48.8
12.8
100.0
12.7
50.4
25.5
49.6
26.5
100.0
26.0
50.6
15.7
49.4
16.3
100.0
16.0
50.1
54.7
49.0
56.1
100.0
55.4
Other
% w/in race
% w/in sex
51.6
76.9
48.4
77.4
100.0
77.1
51.7
52.7
48.3
51.9
100.0
52.3
51.8
70.3
48.2
69.6
100.0
70.0
---
---
---
% w/in race
% w/in sex
51.7
100.0
48.3
100.0
100.0
100.0
51.3
100.0
48.7
100.0
100.0
100.0
51.6
100.0
48.4
100.0
100.0
100.0
50.7
100.0
49.3
100.0
100.0
100.0
Total
Table 5.26 Race by Gender Percentages for CPS vs. NLSY97 Interviews
Race/
Ethnicity
14
CPS Ages 12-16
Gender
Total
Male Female
53.7
46.3
100.0
10.6
9.8
10.2
NLSY97 Total Interviews
Gender
Total
Male
Female
51.4
48.6
100.0
21.3
21.1
21.2
NLSY97 CX Interviews
Gender
Total
Male
Female
51.0
49.0
100.0
13.6
13.7
13.7
NLSY97 SU Interviews
Gender
Total
Male Female
51.7
48.3
100.0
44.5
43.1
43.8
Hispanic
% w/in race
% w/in sex
Non-Hispanic
Black
% w/in race
% w/in sex
51.2
12.5
48.8
12.8
100.0
12.7
50.1
25.4
49.9
26.6
100.0
26.0
49.8
15.6
50.2
16.5
100.0
16.0
50.3
55.4
49.7
56.7
100.0
56.1
Other
% w/in race
% w/in sex
51.6
76.9
48.4
77.4
100.0
77.1
51.6
53.3
48.4
52.3
100.0
52.8
51.6
70.8
48.4
69.8
100.0
70.3
---
---
---
Total
% w/in race
% w/in sex
51.7
100.0
48.3
100.0
100.0
100.0
51.1
100.0
48.9
100.0
100.0
100.0
51.2
100.0
48.8
100.0
100.0
100.0
50.9
100.0
49.1
100.0
100.0
100.0
The percentages in Tables 5.25, 5.26, and 5.27 are slightly outdated due to small changes in the weighted and unweighted totals for the NLSY97 sample after
these tables were produced.
62
Table 5.27 Race by Gender Percentages for CPS vs. NLSY97 Test Completes15
Race/
Ethnicity
15
CPS Ages 12-16
Gender
Total
Male Female
53.7
46.3
100.0
10.6
9.8
10.2
NLSY97 Total Tested
Gender
Total
Male
Female
50.9
49.1
100.0
19.5
19.3
19.4
NLSY97 CX Tested
Gender
Total
Male
Female
51.0
49.0
100.0
11.9
12.0
12.0
NLSY97 SU Tested
Gender
Total
Male Female
50.9
49.1
100.0
44.7
42.3
43.5
Hispanic
% w/in race
% w/in sex
Non-Hispanic
Black
% w/in race
% w/in sex
51.2
12.5
48.8
12.8
100.0
12.7
48.6
24.4
51.4
26.6
100.0
25.5
48.6
15.2
51.4
16.7
100.0
15.9
48.5
55.2
51.5
57.4
100.0
56.3
Other
% w/in race
% w/in sex
51.6
76.9
48.4
77.4
100.0
77.1
51.5
56.0
48.5
54.1
100.0
55.1
51.5
72.9
48.5
71.4
100.0
72.2
---
---
---
Total
% w/in race
% w/in sex
51.7
100.0
48.3
100.0
100.0
100.0
50.6
100.0
49.4
100.0
100.0
100.0
51.0
100.0
49.0
100.0
100.0
100.0
49.5
100.0
50.5
100.0
100.0
100.0
NLSY97 Test Completes include those who completed the full ASVAB battery as well as those who completed the AFQT portion of the ASVAB.
63
Table 5.25 displays comparisons of CPS youths aged 12 to 16 to the screening sample of
NLSY97-eligible youths, overall and by sample type. Because the SU sample consists only of minority
youths, percentages of youths within gender categories are inflated to compensate for the lack of nonHispanic, nonblack youths. This, in turn, inflates the overall within-gender percentages of NLSY97eligible youths; thus, to compare fairly the CPS and the NLSY97, we must concentrate on the CX sample
of NLSY97 eligibles and the within race/ethnicity gender ratios only for the SU sample. Table 5.25
shows that CPS males make up about 51.7 percent and females 48.3 percent of the overall sample. The
NLSY97 CX sample has an almost identical distribution of males and females overall. By race/ethnicity,
the only difference occurs for Hispanics, and this difference is quite small. The CPS shows that 53.7
percent of Hispanics in this age group are males, while the NLSY97 Hispanic, CX sample is 51.5 percent
male. Within race/ethnicity, the supplemental gender ratios are almost identical to the CX sample.
Racial/ethnic percentages within gender show the only noticeable differences. According to the CPS, the
female population of youths is 9.8 percent Hispanic, 12.8 percent non-Hispanic black, and 77.4 percent
other races/ethnicities. The NLSY97 has 14.0 percent Hispanics, 16.3 percent non-Hispanic blacks, and
69.6 percent other races/ethnicities making up the CX sample of females. This suggests that we have a
surplus of Hispanic and non-Hispanic, black youths and an underrepresentation of non-Hispanic,
nonblack youths within the NLSY97 CX sample.
Similarities between CPS and NLSY97 are also evident in Table 5.26, which displays the same
percentage tabulations for NLSY97 interviews. Again, a comparison of the CPS to the NLSY97 CX
sample and the within race/ethnicity gender ratios for the SU sample is appropriate. The results for this
table are almost identical to those for the NLSY97 eligibles. There are very small differences by gender
within race/ethnicity (the CPS shows that 53.7 percent of Hispanics in this age group are males, while the
NLSY97 Hispanic, CX sample is 51.0 percent male and the SU sample is 51.7 percent male), and
somewhat larger racial/ethnic differences within gender categories. For example, the CPS shows that the
female population of youths is 9.8 percent Hispanic, 12.8 percent non-Hispanic black, and 77.4 percent
other races/ethnicities, while the NLSY97 interviewed population has 13.7 percent Hispanics, 16.5
percent non-Hispanic blacks, and 69.8 percent other races/ethnicities making up the CX sample of
females. Again, this suggests that we have a surplus of Hispanic and non-Hispanic, black youths and an
underrepresentation of non-Hispanic, nonblack youths within the NLSY97 CX sample
Finally, Table 5.27 compares the CPS to proportions of NLSY97 youths who completed an
ASVAB test. Overall gender and race/ethnicity distributions are almost identical for the CX sample. The
differences that do occur among subgroups for the CX sample are similar to the NLSY97 eligibles and
interviews; CPS Hispanics have slightly larger proportions of males than the NLSY97. Further, while
CPS non-Hispanic, black youths are 51.2 percent male, the NLSY97 non-Hispanic blacks who completed
a test are only 48.6 percent male. Within race/ethnicity gender ratios for the SU sample are comparable to
results for the CX sample. These small differences between CPS and NLSY97 may be driven by
response rates; females tended to participate at a higher rate than males. Racial/ethnic differences within
gender are smaller than for NLSY97 eligibles and interviews; as stated earlier, the CPS distribution of
females is 9.8 percent Hispanic, 12.8 percent non-Hispanic black, and 77.4 percent other races/ethnicities.
The NLSY97 distribution of CX females who completed a test is 12.0 percent Hispanic, 16.7 percent nonHispanic black, and 71.4 percent other races/ethnicities. Again, this may indicate a surplus of minority
youths and an underrepresentation of non-Hispanic, nonblack youths in the NLSY97 CX sample.
The results shown in these tables reassure us that the NLSY97 sample is balanced
demographically; there is no evidence of a serious differential bias vis-à-vis the CPS. There is no
64
denying that there were shortfalls in overall counts of youths between the ages of 12 and 23; far fewer
youths were found than we had projected. Despite these shortfalls, though, Tables 5.25, 5.26, and 5.27
present evidence that the sample of 12- to 16-year-old youths is balanced and distributed very similarly,
overall and among various subgroups, to the CPS.
Appendices F and G lend more support to these results. The relationship between PSU-level
response rates and PSU-level socio-economic status variables (income and education) is examined in
Appendix F. These analyses indicate only small differences in PSU response rates for PSUs of different
education and income levels. In particular, regression equations suggest a difference of only two percent
in the PSU response rates of high-income/education PSUs and low-income/education PSUs. These
relationships could be muted by the aggregation to the PSU level, since the socio-economic status of
nonrespondents could not be assessed. Appendix G compares family income for NLSY97 youths to the
income of families with resident youths aged 12 to 16 responding to the March 1997 CPS. The results of
these comparisons indicate that while there are some differences in family income between NLSY97 and
CPS youths, the differences are fairly small and are most obvious in the race/ethnicity (particularly
Hispanic) breakouts. Over all of these variables, the NLSY97 sample is reasonably well-balanced in
comparison to the CPS sample.
Examination of NLSY97/PAY97 Coverage Ratios. As stated in Chapter 4, the preliminary
coverage ratios that suggest that “one out of every three NLSY97 eligibles” were missed are very
misleading. The caps used during weighting to prevent extreme weights have a negative effect on the
coverage ratios. But the caps alone do not explain all of the undercoverage of youths observed in
NLSY97.
All surveys suffer from some level of general undercoverage. The presumed true population is
the decennial population adjusted for the census undercount, which is the undercoverage in the census
itself. Estimated coverage rates for the 1990 Census are shown in Table 5.28 below.
65
Table 5.28 Estimated 1990 Census Coverage Ratios (March, 1998)*
Race
Sex
Male
Black
Female
Male
Non-Black
Female
Age
10-19
20-29
10-19
Coverage Ratios
.9805
.9091
.9787
20-29
.9653
10-19
20-29
10-19
20-29
1.0089
.9830
1.0053
.9937
*
These coverage ratios come from Robinson et al. (1993) “Estimation of Population Coverage in the 1990 United States Census
Based on Demographic Analysis,” Journal of the American Statistical Association, Vol, 88, pp. 1061-1071.
Coverage ratios are not given for Hispanics, but Census experts often assume they are similar to
coverage ratios for blacks. Among the age groups of NLSY97/PAY97, this undercoverage is greatest for
black males aged 20-29 -- over 9% were not counted. For most other groups, the undercoverage is small,
and non-black 10-19 year olds actually show overcoverage (i.e., some counted more than once).
The Current Population Survey has greater undercoverage than the Decennial Census. Overall,
the CPS covers about 92 percent of the adjusted decennial-census population. However, the CPS has less
coverage of the age groups of NLSY97/PAY97, as shown in Table 5.29 below.
66
Table 5.29 Estimated CPS Coverage Ratios (March, 1998)
Race
Sex
Male
Age
0-14
15
16-19
20-29
30-39
40-49
0-14
15
16-19
20-29
30-39
40-49
0-14
15
16-19
20-29
30-39
40-49
0-14
15
16-19
20-29
30-39
40-49
0-14
15
16-19
20-29
30-39
40-49
0-14
15
16-19
20-29
30-39
40-49
All Persons
Female
Male
Black
Female
Male
Non-Black
Female
Coverage Ratios
.916
.905
.855
.823
.877
.917
.943
.883
.877
.884
.920
.959
.850
.763
.711
.660
.680
.816
.838
.824
.802
.811
.845
.911
.929
.933
.881
.847
.904
.928
.964
.895
.891
.897
.931
.966
Again, coverage ratios are not given separately for Hispanics. The coverage ratios tend to be
lowest among 20- to 29-year-olds and 16- to 19-year-olds. Undercoverage is most severe for black males
aged 20-29 (.660), which indicates that one in three is not represented. Clearly, the CPS has trouble with
some of the age groups studied by NLSY97/PAY97.
67
Before proceeding, we note that CPS undercoverage is defined by comparing preliminary CPS
estimates of the size of the various demographic domains to the Census Bureau's best demographic
estimates of the sizes of those domains. The final step in CPS weighting actually poststratifies the
preliminary CPS weights to those best estimates of population size. Thus, CPS estimates derived from
final CPS weights exhibit no residual undercoverage relative to the best estimates of population size. We
observe that the CPS age distribution used as the standard of comparison in Figures 5.5-5.7 was created
using the CPS final weights, and thus this age distribution essentially equates to the Census Bureau's best
estimates of population size.
Turning to the NLSY97/PAY97, one way to measure coverage is to look at the ratios of the sums
of weights before and after poststratification. The NLSY97 ratios are given in Table 4.5. However, the
ratios in this table are excessively small because of caps used during weighting (to reduce variability), as
shown in Table 5.18. We believe superior coverage ratios can be derived from the age distribution of
persons screened. We weight up these persons (we used special household weights, see Appendix S) to
get an NLSY97/PAY97 estimate of persons of that age. Dividing these estimates by CPS controls gives
us the estimated coverage ratios in Table 5.30.
Table 5.30 Estimated NLSY97/PAY97 Coverage Ratios
Age Range
0-11
12-16
18-23
24-35
Total
0.90
0.74
0.68
1.01
CX
0.89
0.72
0.69
1.01
SU
0.94
0.83
0.64
1.00
The comparable CPS overall coverage ratios are about .85, so it is clear that NLSY97/PAY97
suffers from more undercoverage than the CPS. Even the overwhelming and costly Decennial Census
suffers from some undercoverage. Large and ongoing samples such as the CPS, however, suffer from
more undercoverage than the Decennical Census.
It is probably reasonable to conclude that both CPS and NLSY97/PAY97 achieved good
coverage of persons age 24-35. For the eligible years, ages 12-23, it may be reasonable to conclude that
both CPS and NLSY97/PAY97 incur a general survey undercoverage of, say, 15 percent. Then,
NLSY97/PAY97 incurs an additional 10 to 15 percent undercoverage attributable to the age-specific
screening which was done. For younger youths age 0-11, CPS misses about 8 percent and the
NLSY97/PAY97 screening sample missed that plus an additional 2 to 3 percent.
Special Studies. We conducted several other special studies to examine the shortfalls in the age
distribution and uncover possible reasons for these shortfalls. As discussed earlier in this report, studies
of language barrier cases, proxies, and gatekeepers have been undertaken as part of the investigation.
Further studies include analyses of crashed cases, noninterview reports, the debriefing of field
interviewers, and an analysis of validation interviews.
68
Appendix D details the analysis of language barrier cases. In this appendix, we focused mainly
on comparing actual PAY97 language barrier cases to national norms for English proficiency. Because
language proficiency is a difficult concept to define and measure, different sources of data arise from
different measurement processes. Thus, the comparability of data sources may be compromised, and
these sources are subject to a variety of differential biases and random errors. Despite these limitations,
the data are presented and compared in this appendix, and there is no evidence that the PAY97 samples
were excessively saturated with language-barrier youths. In fact, these data suggest that the PAY97
samples found language-barrier cases at a typical to below typical rate. We found no support for the
claim that language barrier cases compromise the representativeness of the PAY97 samples to a greater
extent than they compromise the representativeness of other important, national surveys. We believe
these results and conclusions carry over to the NLSY97 samples.
Analyses of proxy and gatekeeper interviews are given in Appendices E and L. Appendix E
summarizes the results of reviewing a statistical sample of the records of calls (ROCs) for cases given a
field disposition indicating that they were screeners completed by proxy respondents. The main purpose
of this analysis was to ensure that the proper procedures were followed and that there were no indications
of youths in the households due to improper use of this disposition code. Of the 5,175 households given
this disposition, NORC reviewed 490. Results show that most of the time (92.2 percent), field
interviewers correctly followed procedures for proxy screeners. In those cases where procedures were not
followed, only 0.6 percent were households with possible youths present. The analysis of gatekeeper
interviews in Appendix L found similar results. This report summarizes the results of reviewing a
statistical sample of ROCs for cases coded as gatekeeper interviews. Again, the purpose of the analysis
was to verify that proper procedures had been followed. NORC reviewed 494 of the 4,055 households
identified as gatekeepers. Again, the majority of the time (78.5 percent), field interviewers correctly
followed procedures; in the cases when procedures were not followed, only 0.6 percent were households
with possible youths present. We conclude that gatekeeper interviews may have provided a minor
contribution to the hole in the age distribution, but that they could not be a major source of the hole.
NORC debriefed 96 of the 432 NORC field interviewers who had administered screening
interviews for NLSY97/PAY97, and an analysis of these debriefing interviews is given in Appendix O.
The debriefing interviews found, among other things, that the household informants may have discovered
the eligible age range through various sources and then either misreported their youths’ ages or failed to
report eligible youths at all. For example, the NLSY97 and PAY97 brochures and letters, which were
sent to households prior to screening (see Appendix K), disclosed information about the age range or, at
least, the study’s focus on youths. These materials also disclosed the amount of time required of
participants, requesting the youth to “spend approximately one hour with the interviewer…” and “take a
2-hour test on a computer…” This may have discouraged participation, possibly resulting in purposeful
underreporting of youths or misreporting of resident youths’ ages to avoid participation in the study.
Also, neighbors who had already been screened may have supplied the relevant information to other
informants, giving them the opportunity to misreport their youths. Finally, the field interviewers
themselves may have given this information to respondents during the screening process. In fact, 38 of
the 96 interviewers (nearly 40 percent) reported telling informants the ages of youths eligible to
participate in NLSY97/PAY97. Twenty-two of the interviewers, about 23 percent, said that they
followed the intended script -- presented to them at training sessions -- during the screening interviews
(for examples of the script, see Appendices H and I).
69
The debriefing interviews also examined possible computer problems involved in screening.
Seventeen of the 96 interviewers conducted only paper and pencil screening interviews, while the
remaining 79 used the laptop computer for screening. Of the 79, 32 reported experiencing problems with
the computer, and 26 interviewers reported that computer crashes which prevented the completion of a
screening interview occurred at least once at a household where they thought there were eligible youths.
However, this analysis concluded that although the interviewers reported problems with the computers,
the frequency of the problems was not high; therefore, computer problems alone could not explain the
missing youths in the sample.
A thorough analysis into the computer crashes that occurred during screening can be found in
Appendix N. Here, reasons for the crashes, including low random access memory (RAM), insufficient
space on the hard drive, inconsistent battery life, and incorrectly assigned cases, were cited. The four
PSUs with the lowest response rates were examined, and final dispositions for each case within those
PSUs were analyzed. The results show that in these four PSUs, the large number of crashes and other
technical problems, ranging from 22.5 to 64.0 percent of total cases, could have contributed to the
shortfalls in the eligible age range. However, nationwide there were a total of 1,318 crashed cases, 357 of
which fell into those four PSUs. The remaining 961 average to only 4.9 cases per PSU for the other 196
PSUs. Thus, it is unlikely that, in the remaining PSUs, computer problems were major contributors to the
shortfalls in the age distribution.
Appendix M documents the incidence of noninterviews in the NLSY97 screening effort and
examines the data from the noninterview reports (NIRs). For each of the noninterview households, the
NORC field interviewer assigned to the case was asked for his or her opinion about the makeup of the
household, including how many people they thought lived in the household, overall and in particular age
categories (10-25 and 17-25). The data collected from these reports indicated that 4,289 people were
presumed to be living in the noninterviewed households. 1,401 household members were reported to be
between the ages of 10 and 25, and 570 between the ages of 17 and 25. The subtraction of youths in the
latter age group from the first gives a rough approximation of the number of youths aged 10 to 16;
according to this calculation, 831 youths fall into that age range. In other words, 19.4 percent of the
people in noninterviewed households were reported to be between the ages of 10 and 16, and 13.3 percent
between 17 and 25 years of age; these rates are two to three times what we would expect in the nation as a
whole. The NLSY97/PAY97 found that only 6.1 percent of people (in screened households) were
between ages 12 and 16, and only 5.5 percent were between 18 and 23. The March 1997 CPS estimates
are only slightly higher; 7.3 percent of people were aged 12 to 16, and 7.9 percent were aged 18 to 23.
The fact that youths in similar age groups comprised a much higher percentage of the population in
noninterviewed households than in the screened households or in the overall population provides
evidence that households with youths were more likely to refuse to be screened, thus contributing to the
hole in the NLSY97/PAY97 age distribution.
An analysis of validation interviews, conducted in an effort to help explain the undercount of
youths in the NLSY97/PAY97 eligible age range, is given in Appendix P. The validation process entails
reinterviewing a subsample of the households that were screened in the NLSY97; its main objectives are
for the respondents and NORC managers to evaluate the performance of the interviewers. A total of
1,296 validation interviews were conducted on respondents who were originally interviewed between
January and October 1997, including 347 screeners and 949 youth and parent interviews. A crosstabulation of interview dates and validation interview dates showed that the majority of the validation
70
interviews occurred at least one month after the original interview. In the comments section of the
interview, 338 respondents remarked on the interviewers’ mannerisms. Of these, 80.5 percent reported
that the interviewers were polite and courteous, 7.4 percent were neutral, and the remaining 12.1 had
negative comments. The analyses of the validation interviews failed to reveal any poor conduct on the
part of the interviewers, as most of the comments of the respondents were positive and the few negative
comments pointed to circumstances beyond the control of the interviewers. On the other hand, the field
staff failed to conduct timely validation interviews, which leave the results somewhat inconclusive.
Finally, in Appendix T, we compared the NLSY97 age distributions of various groups based on
whether they live in urban centers, suburbs, other urban areas, or rural areas. The age distributions for all
groups have a drop in the number of people in the 12-23 age range. The urban age distribution shows a
gradual drop in percentages of people in the entire age range, while the rural age distribution shows a
sharper drop for older youths. For the NLSY97, we are primarily interested in the numbers of 12- to 16year-olds. This group appears to be undercounted in urban rather than in rural areas. However, we
cannot attribute the undercount to one particular type of urban area. A comparison of the CPS and
NLSY97 age distributions indicate that the undercount is larger for older youths in rural areas and small
cities, larger for younger youths in urban centers, and uniform across the 12-23 age range in suburban
areas. These comparisons should be interpreted cautiously, though; we are using the overall CPS as a
benchmark, since the CPS does not allow for breakdowns by different rural/urban groups. Thus, our
analysis of rural/urban age distributions does suggest that there are some differences in the shapes of the
distributions and undercounts of NLSY97 youths for different types of areas. However, there is no
compelling evidence which would allow us to attribute the undercount to one particular type of
rural/urban category.
5.4
Estimates of the NLSY97 Shortfall
In previous sections and in various appendices, we have described a shortfall of eligible youths
age 12-16 in the NLSY97 screening sample. In this section, we produce our best estimate of the size of
the shortfall, defined as the number of eligibles expected minus the number of eligibles found in
screening operations. While the number of eligibles found is known, the number of eligibles expected
must be estimated. Expectations, of course, depend upon conditioning variables, if any. More on this
critical factor later.
We have determined and now present five different methods of estimating the number of eligibles
expected. In brief, they are as follows:
Method 1A uses the March 1997 Current Population Survey (CPS) to determine a rate of eligible
youths per household by race/ethnicity. The SU sample is assumed to find youths at a higher rate
than the CX sample. The method multiplies the rates by the number of completed screeners to
estimate the number of eligibles expected.
Method 1B is identical to Method 1A, except it makes an alternative assumption about the
increased success of the SU sample in finding youths.
Method 2 employs the 1990 Decennial Census totals of youths age 0-17 by race/ethnicity for
each NLSY97 segment to calculate an estimate of the number of eligibles expected.
Method 3A uses the March 1997 CPS to determine a rate of eligible youths per household by
71
age. The SU sample is assumed to find youths at a higher rate than the CX sample. The method
multiplies the rates by the number of completed screeners to estimate the number of eligibles
expected.
Method 3B is identical to Method 3A, except it makes an alternative assumption about the
increased success of the SU sample in finding youths.
In what follows, we describe each of these methods in greater detail, and we implement the methods to
estimate the shortfall.
We also determined and implemented two additional methods (Method 4A and 4B) of estimating
the number of eligibles expected, both of which are based upon the distributions of HHs by number of
youths, as presented in Table 5.19. Because these methods give similar results to the methods presented
here, we omit discussion of them from this report.
For the cross-sectional (CX) sample, Method 1A is straightforward. Table 5.31 shows the actual
field numbers, following the sample all the way from the number of addresses fielded, to the number of
completed screeners, to the number of eligible youths. It also shows three different estimates of the
shortfall. The first estimate uses the expected sample sizes given in the original sampling plan for the
NLSY97. The second and third estimates use expectations based upon the March 1993 CPS rates (used
for the original sampling plan) and the March 1997 CPS rates (closest in time to the NLSY97 fieldwork)
applied to the actual number of screened households. The last column gives the source table for each of
the actual field numbers.
Unfortunately, Method 1A is much more difficult to apply to the supplemental (SU) sample. The
main difficulty is that CPS rates are not suitable for the SU sample. The oversampling done in the SU
sample implies higher youth rates, especially for Hispanic and black youths, than those available from an
essentially self-weighting survey like the CPS.
72
Table 5.31 Analysis of Actual versus Expected Youths in CX Screening Sample: Method 1A
Addresses fielded
Proportion eligible for
screening
Addresses eligible for
screening
Response rate for
screening
Completed Screeners
Expectations
from the
Original
Sampling
Plan
64,654
0.85
Expectations
(March 93
Expectations
CPS) Applied (March 97 CPS)
to Actual
Applied to
Number of Actual Number
Screened HHs of Screened HHs
65,269
65,269
0.882
0.882
Actual
Field
Numbers
65,269
0.882
Source for
Actual Field
Numbers
Table 2.2
54,956
57,562
57,562
57,562
0.91
0.943
0.943
0.943
50,010
54,253
54,253
54,253
Table 5.1
Number of eligible youths per screened household
Hispanic
0.021
0.021
Non-Hispanic black
0.028
0.028
Non-Hispanic nonblack
0.131
0.131
TOTAL
0.180
0.180
0.024
0.029
0.137
0.190
0.019
0.022
0.095
0.136
Calculated
Calculated
Calculated
Sum
1,026
1,175
5,134
7,335
Table 3.3
Table 3.3
Table 3.3
Sum
Number of eligible youths
Hispanic
Non-Hispanic black
Non-Hispanic nonblack
TOTAL
1,050
1,400
6,551
9,001
1,139
1,519
7,107
9,765
1,302
1,573
7,433
10,308
Shortfall of eligible youths
Hispanic
Non-Hispanic black
Non-Hispanic nonblack
TOTAL
24
225
1,417
1,666
113
344
1,973
2,430
276
398
2,299
2,973
Table 5.1
We performed simulations during planning stages of the NLSY97 and estimated that the
oversampling would double the percentage of youths that would be Hispanic or black. In the actual
survey, we found the percentages more than doubled. The percentage of 12- to 16-year-olds in the SU
who were Hispanic was 30.22 percent, which is 2.59 times the national rate of 11.67 percent. Also, the
percentage of 12- to 16-year-olds in the SU who were non-Hispanic black was 37.80 percent, which is
2.43 times the national rate of 15.55 percent. Therefore, the oversampling was even more efficient than
the simulations suggested it would be. However, these simulations did not reflect one layer of
oversampling that was added during actual sampling operations. This layer was the selection of housing
units in “high” minority segments at ten times the rate of housing units in “low-low” minority segments.
This additional layer could be the reason the SU was even more efficient in finding minority youths than
the simulations suggested.
73
Despite the difficulties, we have made estimates for the number of eligible youths per HH by
race/ethnicity using the following assumptions:
1. The CPS total number of eligible youths per household is .180 (1993) or .190 (1997).
2. We found 27.2 percent more NLSY97 youths per household in the SU (3,638 eligibles in 21,157
completed screeners) than in the CX (7,335 eligibles in 54,253 completed screeners).
3. Therefore, we assume for Table 5.32 that the total number of eligible youths per household in the SU
is .180 x 1.272 = .229 for the 1993 CPS expected values and .190 x 1.272 = .242 for the 1997 CPS
expected values.
4. The SU NLSY97 sample consisted of 1,102 eligible Hispanics (30.2 percent), 1,371 eligible nonHispanic blacks (37.8 percent), and 1,165 eligible non-Hispanic nonblacks (32.0 percent).
5. Therefore, for the 1993 CPS column, we assume the numbers of eligible youths per household are
.229 x .302 = .069 (Hispanic), .229 x .378 = .087 (non-Hispanic black), and .229 x .320 = .073 (nonHispanic, nonblack). Similar numbers for the 1997 CPS are .073, .091, and .077.
Of course, assumption 3 is questionable because it assumes the shortfall occurred at the same rate in the
CX and SU samples. Assumption four is based upon the premise that the shape of the race/ethnicity
distribution is the same for the undercovered SU cases as for the covered cases.
Given these assumptions, Table 5.32 shows estimates of the shortfalls of eligible youths in the SU
sample, using the same format as Table 5.31. One additional column has been added because 20 percent
of the SU sample represented in the original sampling plan was not fielded. The extra column represents
information based upon the actual number of released addresses.
Table 5.33 then shows the same information as Tables 5.31 and 5.32 for both samples combined.
The estimated shortfall, including SU non-Hispanic nonblacks, is 4,433 youths. Including only NLSY97
eligibles (excluding SU non-Hispanic nonblacks), the estimated shortfall is 3,969 youths.
74
Table 5.32 Analysis of Actual versus Expected Youths in SU Screening Sample: Method 1A
Addresses fielded
Proportion eligible
for screening
Addresses eligible
for screening
Response rate for
screening
Completed
Screeners
Expectations
from the
Original
Sampling
Plan
31,855
0.85
Expectations
from the Expectations Expectations
Original
(March 93
(March 97
Sampling CPS) Applied CPS) Applied
Source for
Plan,
to Actual
to Actual
Actual
Actual
Released
Number of
Number of
Field
Field
Addresses Screened HH Screened HH Numbers Numbers
25,485
25,688
25,688
25,688 Table 2.2
0.85
0.881
0.881
0.881
27,077
21,662
22,642
22,642
0.91
0.91
0.934
0.934
24,640
19,713
21,157
21,157
21,157 Table 5.1
0.069
0.087
0.073
0.073
0.091
0.077
0.052 Calculated
0.065 Calculated
0.055 Calculated
0.229
0.241
0.172
1,102 Table 3.3
1,371 Table 3.3
1,165 Screening
data
3,638
Sum
Number of eligible youths per screened household1
Hispanic
0.041
0.041
Non-Hispanic black
0.057
0.057
Non-Hispanic
0.082
0.082
nonblack
TOTAL
0.180
0.180
Number of eligible youths1
Hispanic
Non-Hispanic black
Non-Hispanic
nonblack
TOTAL
1,010
1,404
2,020
808
1,124
1,616
1,460
1,841
1,544
1,544
1,925
1,629
4,434
3,548
4,845
5,098
22,642 Table 5.1
0.934
Sum
Shortfall of eligible youths
Hispanic
-92
-294
358
442
Non-Hispanic black
33
-247
470
554
Non-Hispanic nonblack
855
451
379
464
TOTAL
796
-90
1,207
1,460
1
This also includes non-Hispanic nonblack youths screened in the SU sample.
The estimates in Tables 5.32 and 5.33 were based fundamentally on the assumption (derived from
the actual NLSY97 yields) that the SU screening sample would to yield 27.2 percent more youths per
household than the CX screening sample. An alternative assumption can be derived from 1990 Decennial
Census data for NLSY97 segments. Using this alternative, Table R-1 shows that we would expect 10,413
eligible youths from 54,253 CX screeners and 5,552 eligible youths from 21,156 SU screeners. This SU
rate is 36.72 percent larger than the CX rate. Table R-1 also shows that 26.85 percent (1,491/5,552) of
the SU expected youths are Hispanic, 36.91 percent (2,049/5,552) are non-Hispanic black, and 36.24
percent (2,012/5,552) are non-Hispanic, nonblack. Combining these numbers leads to larger estimates for
the rates of eligible youths per household. For the 1993 CPS, we have .180 x 1.3672 x .2685 = .066
75
(Hispanic), .180 x 1.3672 x .3691 = .091 (non-Hispanic black), and .180 x 1.3672 x .3624 = .089 (nonHispanic, nonblack). Similar estimated rates for the 1997 CPS are .070 (Hispanic), .096 (non-Hispanic
black), and .094 (non-Hispanic, nonblack). These larger rates, of course, lead to larger estimates of the
shortfall, called Method 1B, that appear in Table 5.36.
Table 5.33 Analysis of Actual versus Expected Youths in NLSY97 Screening Sample
(CX + SU): Method 1A
Addresses fielded
Proportion eligible for screening
Addresses eligible for screening
Response rate for screening
Completed Screeners
Expectations from
Expectations
the Original (March 93 CPS)
Sampling Plan, Applied to the
Released Actual Number
Addresses of Screened HH
90,139
90,957
0.85
0.882
76,618
80,204
0.91
0.940
69,723
75,410
Number of eligible youths per screened household2
Hispanic
0.027
Non-Hispanic black
0.036
Non-Hispanic nonblack
0.117
TOTAL
0.180
Number of eligible youths2
Hispanic
Non-Hispanic black
Non-Hispanic nonblack
TOTAL
1,858
2,524
8,167
12,549
Expectations
(March 97
CPS) Applied
to the Actual
Number of Actual Field
Screened HH
Numbers
90,957
90,957
0.882
0.882
80,204
80,204
0.940
0.940
75,410
75,410
0.034
0.045
0.115
0.194
0.038
0.046
0.120
0.204
0.028
0.034
0.084
0.146
2,599
3,360
8,651
14,610
2,846
3,498
9,062
15,406
2,128
2,546
6,299
10,973
Shortfall of eligible youths
Hispanic
-270
471
Non-Hispanic black
-22
814
Non-Hispanic nonblack
1,868
2,352
TOTAL
1,576
3,637
TOTAL ELIGIBLES
1,125
3,258
2
This also includes non-Hispanic nonblack youths screened in the SU sample.
718
952
2,763
4,433
3,969
Method 2 is explained in detail in Appendix R. In this appendix, expected counts of youths age
12-16 were estimated using the actual 1990 Decennial Census counts for the 1,751 NLSY97 segments.
As observed moments ago in our presentation of Method 1B, Table R-1 shows the number of youths age
12-16 that we expected to screen. Table R-2 shows the actual counts of youths screened, and Table 5.34
below shows the differences between these two tables. Therefore, the estimate of the NLSY97 shortfall,
including SU non-Hispanic nonblacks, using this method is 4,992 youths. Including only NLSY97
eligibles, the estimate is 4,145 youths. These numbers are very similar to those from Method 1B, which
we would expect because both methods used relationships between the CX and SU samples observed in
76
1990 Census data.
Table 5.34 Estimates of the Shortfall of Youths Aged 12-16 in the NLSY97
Screening Sample: Method 2
Sample
CX
SU
TOTAL
Hispanic
138
389
527
Non-Hispanic,
Black
370
678
1,048
Non-Hispanic,
Nonblack
2,570
847
3,417
TOTAL
3,078
1,914
4,992
Method 3A simply uses the age distributions from the CPS and NLSY97, comparing expected
counts using the March 1997 CPS to actual eligible counts from the NLSY97. In calculating expected
counts, we first found the average rate of persons per household by single year of age in the CPS (using
weighted data). For example, the rate for 12-year-olds per CPS household is 0.0391. Next, the number of
NLSY97 screened households was multiplied by each of the age specific rates to determine the number of
people of each age expected to be found within the screened households, according to the CPS rates. This
method was used for the CX sample, but the additional factor derived for Method 1A of 1.272 was used
for the SU sample. This factor of 1.272 reflects the fact that 27.2 percent more eligible youths per
household were found in the SU sample than in the CX sample. The method requires the additional
assumption that the shape of the CPS age distribution applies to both the CX and SU samples.
These calculations result in a set of expected numbers of persons by single year of age for the CX
and SU samples. For example, the CX number of screened households (54,253) multiplied by the CPS
rate of 12-year-olds per household (0.0391) yields an expected 2,123 youths age twelve. For the SU
sample, the expected number of youths age twelve is 21,157 x 1.272 x .03913 = 1,053. Finally, actual
NLSY97 numbers of persons screened were compared to the expected numbers to identify patterns and
differences in various age ranges. The Method 3A estimates appear in Table 5.35.
Table 5.35 Expected Youths versus Actual Youths Screened: Method 3A
Age
12
13
14
15
16
Total
March Expected Expected Expected Actual Actual Actual
1997
Total
CX
SU Total
CX
SU
CPS Sample Sample Sample Sample Sample Sample
Rate
0.03913
3,177
2,123
1,053 2,139 1,435
704
0.03773
3,063
2,047
1,016 2,179 1,504
675
0.04028
3,269
2,185
1,084 2,229 1,494
735
0.03811
3,094
2,068
1,026 2,253 1,498
755
0.03860
3,134
2,094
1,040 2,173 1,404
769
0.19385
15,737
10,517
5,219 10,973 7,335
3,638
77
Shortfall Shortfall Shortfall
Total
CX
SU
Sample Sample Sample
1,037
884
1,040
841
961
4,763
688
543
691
570
690
3,182
349
341
349
271
271
1,581
We estimate a total shortfall of 4,763 youths using this method. Please note that the totals in
Table 5.35 include all 12- to 16-year-olds, even the non-Hispanic nonblacks in the SU sample who were
not eligible for the NLSY97. This method does not yield an estimate of the shortfall of eligible youths
because it divides the shortfall by age rather than by race/ethnicity. In theory, we could devise yet
another method of estimating the shortfall via CPS youth rates by age by race/ethnicity. However, time
prevented us from pursuing this approach.
Recalling Method 1B, we can obtain alternative estimates of expected youths by substituting
1.3672 for 1.272 as the increase in youths per household associated with the SU sample. This method -which we call Method 3B -- leads to an expected SU sample size of 5,610 (total sample size of 16,127),
and an estimated SU shortfall of 1,972 (total shortfall of 5,154). This estimate of the total shortfall is very
close to the estimates from Method 1B and Method 2.
Table 5.36 Five Estimates of the Shortfall of Eligible Youths
SU youth rate increase
Shortfall of eligible youths, CX
Hispanic
Non-Hispanic black
Non-Hispanic nonblack
CX TOTAL
Shortfall of eligible youths, SU
Hispanic
Non-Hispanic black
Non-Hispanic nonblack
SU TOTAL
SU TOTAL ELIGIBLE
Shortfall of eligible youths,
TOTAL
Hispanic
Non-Hispanic black
Non-Hispanic nonblack
TOTAL
TOTAL ELIGIBLE
Method 1A Method 1B
27.2%
36.72%
Method 2 Method 3A Method 3B
36.72%
27.2%
36.72%
276
398
2,299
2,973
276
398
2,299
2,973
138
370
2,570
3,078
442
554
464
1,460
996
379
660
824
1,863
1,039
389
678
847
1,914
1,067
718
952
2,763
4,433
3,969
655
1,058
3,123
4,836
4,012
527
1,048
3,417
4,992
4,145
3,182
3,182
1,581
1,972
4,763
5,154
Table 5.36 displays estimates for all five methods. All methods suggest a similar level of
shortfall in the CX sample. It seems clear from these data that a critical factor is the assumed increase in
youths per household applicable to the SU sample. The three methods that employ the 36.72-percent
assumption all give similar estimates of the total shortfall (including SU non-Hispanic, nonblack youths).
And the two methods, Methods 1A and 3A, that employ the 27.2-percent assumption, also give similar
and somewhat lower estimates of the shortfall.
78
A reasonable question one might ask is why there is any increase at all. The main reason lies in
how the SU sample was selected. It was an oversample not just of minorities (Hispanics and nonHispanic blacks), but of youths. The selection probabilities for PSUs and segments were proportional to
their numbers of minority youths (according to the 1990 Census). Therefore, we expect the SU to be
denser not just in minorities, but in youths.
In our opinion, however, the 36.72 percent higher rate, as shown by 1990 Census data, is likely to
have been smaller by the time of NLSY97 fielding due to the regression effect. Mobility in the
population will change minority youth densities positively in some areas and negatively in others, but in
the specific NLSY97 segments chosen because of their high minority youth density, the effect is likely to
be negative. Therefore, the actual 1997 SU increase in youths per household above the CX rate is
unknown, but is likely to be less than 36.72 percent. This implies that any estimate of shortfall using the
36.72-percent assumption will be conservative (in the sense of tending to be an overestimate of the
shortfall).
Methods 1B, 2, and 3B also have another undesirable property, namely that the assumed
race/ethnicity distribution used in calculating youth rates for the SU sample is borrowed from 1990
Census data. Thus, the estimates ignore the tremendous growth in the Hispanic population between 1990
and 1997.
Another reasonable estimate of the increase is 27.2 percent, which is the observed NLSY97
increase. This estimate arises under the assumption that the shortfall affects the CX and SU samples
equally, an assumption that we cannot conclusively verify nor exclude on the basis of our data.
On balance, considering the regression effect and other evidence presented in this overall report,
we prefer the factor 1.272 rather than 1.3672, and thus we prefer Methods 1A and 3A.
In theory, Methods 1A and 3A should probably agree more closely than they do. The observed
differences between them arise almost entirely from differences between the 1997 CPS youth rates
employed in Tables 5.31 (0.190) and 5.35 (0.19385). The youth rates should agree but do not because
they were produced by different analysts working at different points in time, using slightly different
assumptions. The difference between the youth rates is quite small, but when multiplied by the 21,157
HHs in the SU sample, even a small difference in rates implies an absolute difference of over 300 youths.
We believe Method 1A is as reasonable as Method 3A, and it has the added virtue that it yields
estimates of eligible youths. Thus, we eliminate Method 3A from further discussion.
If the rate of shortfall in the SU sample is smaller than in the CX sample, then even Method 1A
would tend towards an overestimate of the shortfall, and vice versa. Normally, in census and survey work
we expect higher undercount rates for Hispanics and blacks than for non-Hispanic nonblacks. If this
effect applies to the NLSY97, then we would be forced to conclude that the SU shortfall rate is higher
than the CX rate, and thus that Method 1A underestimates the shortfall. On the other hand, in this report
we have reviewed evidence that suggests higher undercount rates for non-Hispanic nonblacks and
generally for affluent domains. If this effect is, in fact, real, then we would conclude that the SU shortfall
rate is the smaller rate and that Method 1A overstates the shortfall. On balance, we would characterize
this evidence as somewhat weak, however, and given the evidence before us, we are reluctant to deviate
far from the 1.272 assumption.
79
Thus, in our opinion, Method 1A supplies the preferred estimates of the shortfall in the NLSY97.
After rounding, we estimate the screening samples missed a total of 4,400 youths age 12-16, and a total of
4,000 eligible youths, relative to the hypothetical true population. Dividing estimated shortfalls for total
youths by expected total youths gives the following shortfall rates:
TOTAL
Hispanic
Non-Hispanic, black
Non-Hispanic, nonblack
28.7 percent
25.2 percent
27.2 percent
30.5 percent.
These rates are apparently quite consistent with the coverage ratios presented in Table 5.30 – both suggest
the NLSY97 coverage was less than 75 percent.
Finally, at the opening of this Section 5.4, we mentioned but postponed an examination of the
effect of conditioning on estimated shortfalls. Let us now address this important issue. The estimated
shortfalls studied here using Methods 1A, 1B, 2, 3A, and 3B are conditional on (1) the sizes of the
screening samples and (2) the assumption that the screening samples comprise representative probability
samples of the population of in-scope American HHs. The first conditioning factor is obvious, because
all of the expected numbers of youths were built up from various youth rates applied to the actual number
of HHs screened. The second conditioning factor is required because the expected numbers of youths
were built up from CPS or Census youth rates, which we take to be representative of the population of inscope American HHs. Therefore, let us pose the following hypothetical question:
Suppose new, representative screening samples were fielded, according to the NLSY97
design, resulting in 54,253 and 21,157 completed screening interviews in the new CX and SU
samples, respectively. Further, suppose the screener response mechanism in these new
samples is fully ignorable (i.e., response is unrelated to the presence of youths in the HH).
How many youths age 12-16 would the new screening samples expect to find, and what is the
shortfall of youths in the actual NLSY97 screening samples vis-à-vis the new, ideal screening
samples?
The methods and estimated shortfalls studied here provide appropriate answers to this question. For later
{
}
use, let y denote the count of youths in the actual sample and let E Y | n, f ignorable denote the expected
sample size as defined in this hypothetical question.
On the other hand, through our work in Sections 5.1-5.3 and various appendices, especially
Appendix M, we strongly believe the actual NLSY97 screening samples encountered a nonignorable
response mechanism (i.e., response is related to the presence of youths in the HH). Our belief is that HHs
with no eligible youths responded at a higher rate than HHs with youths, and HHs with youths had a
lower propensity to respond to the screening interviews. Therefore, let us pose a second hypothetical
question:
Given the actual numbers of screening interviews (54,253 and 21,157) and the type of
nonignorable response mechanism we suspect applies to the actual NLSY97 screening
samples, how many eligible youths were expected and what is the shortfall of the actual
youths screened vis-à-vis this expectation?
80
This question seems highly relevant to us, yet the methods and estimated shortfalls studied here do not
provide an answer. In fact, we have no solid information on which to formulate a quantitative answer to
this question.
Had we been able to estimate the shortfall given the screening sample sizes and the nonignorable
response mechanism, we would be in the enviable position of being able to partition the shortfall into
{
}
components due to the response mechanism and to misreporting. Let E Y | n, f nonignorable denote the
expected sample size as defined by this second hypothetical question.
decomposed as follows:
{
= E{Y | n, f
}
} − E{Y|n, f
Then, the shortfall can be
Total shortfall = E Y | n, f ignorable − y
ignorable
nonignorable
} + E{Y|n, f
nonignorable
}−y
= Shortfall due to response mechanism + Shortfall due to misreporting.
As it stands, this decomposition is unobservable given the evidence we have available to us.
5.5
Estimates of Variance
Two types of errors -- sampling and nonsampling--affect sample survey statistics. The standard
errors calculated for the NLSY97 reflect, for the most part, the magnitude of the sampling error. They
also take into account some of the effects of nonsampling error related to response and enumeration, but
they do not account for any systematic biases in the data.
Nonsampling error in surveys can be attributed to a variety of sources, including inability to
obtain information about all persons in the sample, differences in question interpretation, inability or
unwillingness of respondents to provide accurate information, inability to recall information, processing
errors, imputation errors, and undercoverage. Several studies were undertaken to investigate the existence
and prevalence of nonsampling error. Undercoverage was discussed extensively throughout this chapter,
and several special studies were discussed in Section 5.3 and are described in detail in Appedices D, E, L,
M, N, O, and P.
Sampling error is the name given to the between-sample variation in sample-based estimates.
These differences occur by chance, and the variability is measured by the standard errors of the
estimates. Sample estimators from a given survey design are unbiased when an average of the estimates
from all possible samples would yield the true population value. In this case, the sample estimate and its
estimated standard error can be used to construct approximate confidence intervals, or ranges of values,
that include the true population value with known probabilities. Specifically, approximately 95 percent of
the intervals from two standard errors below the estimate to two standard errors above would include the
true population parameter.
81
The impact of departures from simple random sampling on the precision of sample estimates is
often measured by the design effect (DEFF). The design effect is defined as the ratio of the variance
corrected for the sampling design to the variance that would be obtained given a hypothetical simple
random sample. Most complex, multi-stage designs result in a design effect greater than one; in other
words, the variance of an estimator is greater than the variance that would be obtained had the data been
based on a simple random sample.
To estimate the variance using information about the sample design, it is necessary to use
statistical procedures such as Taylor series approximations, Balanced Half Samples, or Jackknife. For the
NLSY97, NORC used the Taylor series procedure to calculate standard errors. Finally, the square root of
the design effect, referred to as DEFT, is another useful measure provided for multiplication by simple
random sample standard errors to obtain design-corrected standard errors.
Standard errors, design effects, and root design effects were calculated for six domains in the
NLSY97 sample, including the full sample, males, females, Hispanics, non-Hispanic blacks, and other
races/ethnicities, and for a total of 25 binary and continuous survey variables. They can be found in
Tables 5.37 through 5.48. For the binary variables (odd-numbered tables), the sample size estimate (in
percent), design and simple random sample standard errors (in percent), and design effect and root design
effect are displayed. For the continuous variables (even-numbered tables), the total, mean, design and
simple random sample standard errors of the mean, and design effect and root design effect are given.
Summaries of the design effect and root design effect are also given for each set of variables in each
domain, including the mean, standard deviation, minimum, median, and maximum.
A final table, Table 5.49, summaries the root design effects among the six domains two types of
variables. As a rough rule of thumb, we recommend that NLSY97 analysts divide the statistical test
statistics, derived under simple random sampling assumptions, by the median DEFT (or by its square in
the case of χ 2 tests) prior to consulting the reference distributions of the tests and to determining the pvalues. This practice will approximately correct the test statistics for the fact that the NLSY97 is not
based on a simple random sample, but instead is based upon correlated observations.16
16
For more details, see Skinner, C.J., Holt, D., and Smith, T.M.F. (1989), Analysis of Complex Surveys, John Wiley
& Sons, New York.
or
Lehtonen, R. and Pahkinen, E.J. (1994), Practical Methods for Design and Analysis of Complex Surveys, John
Wiley & Sons, New York.
82
Table 5.37 Variance Estimates for Overall NLSY97 Sample: Proportions
Proportions (Binary Variables)
Sample
Size
8,983
Estimate (in
Percent)
97.45
Design SE
(in Percent)
0.21
SRS SE (in
Percent)
0.17
DEFF
1.24
DEFT
1.11
R in labor force?
8,984
17.49
0.57
0.40
1.42
1.19
R employed?
8,984
17.17
0.55
0.40
1.38
1.17
R working >= 10 hrs/week? (alternative 1)17
8,837
9.15
0.38
0.31
1.22
1.11
R working >= 10 hrs/week? (alternative 2)18
8,837
11.99
0.45
0.34
1.32
1.15
R employed during school year?
8,984
10.58
0.47
0.32
1.47
1.21
R ever worked during school year?
8,984
19.77
0.58
0.42
1.38
1.18
R employed during summer?
8,984
12.20
0.46
0.34
1.35
1.16
R ever worked during summer?
8,984
15.99
0.53
0.39
1.36
1.16
R ever retained a grade?
7,708
15.01
0.70
0.41
1.71
1.31
R ever skipped a grade?
7,718
1.84
0.19
0.15
1.27
1.12
R ever in remedial classes?
3,872
38.93
1.15
0.78
1.47
1.21
R have a computer at home?
5,410
58.13
1.19
0.67
1.78
1.33
R have a quiet place to study?
5,408
90.14
0.48
0.40
1.20
1.10
R’s parent born in US?
7,936
88.63
0.91
0.36
2.53
1.59
R’s parent has worked at least 3 months since R
born?
DEFF/DEFT Summary
7,927
95.00
0.53
0.24
2.21
1.49
Mean
1.52
1.22
Standard Deviation
0.37
0.14
Minimum
1.20
1.10
Median
1.38
1.18
Maximum
2.53
1.59
R enrolled?
17
For this variable, all youths who reported working at job(s) more than 10 hours per week were coded as 1; youths reporting working less than
10 hours per week were coded as 0; and youths reporting “hours vary” were coded as 0. An alternative to this coding scheme is given in the
following variable and described in the footnote below.
18
For this variable, all youths who reported working at job(s) more than 10 hours per week were coded as 1; youths reporting working less than
10 hours per week were coded as 0; and youths reporting “hours vary” were coded as 1.
83
Table 5.38 Variance Estimates for Overall NLSY97 Sample: Means
Continuous Variables
Number of weeks worked in 1996?
Sample Size
5,398
Estimated
Mean
8.11
Design SE
0.30
SRS SE
0.20
DEFF
1.50
DEFT
1.22
Days/wk have dinner with family?
5,356
5.19
0.04
0.03
1.33
1.15
Days/wk have fun with family?
5,356
2.67
0.03
0.03
1.00
1.00
Hours/week on homework?
4,723
5.99
0.10
0.07
1.43
1.20
Hours/week watching TV?
5,382
17.26
0.25
0.17
1.47
1.21
Hours/week reading for pleasure?
5,382
2.92
0.08
0.06
1.33
1.15
Parent’s income
5,211
22,474.78
443.48
261.88
1.69
1.30
Spouse’s income
3,797
37,698.98
991.44
599.69
1.65
1.28
Percent chance of R getting 4-year degree by age
30
DEFF/DEFT Summary
3,004
69.73
0.75
0.59
1.27
1.13
Mean
1.41
1.18
Standard Deviation
0.21
0.09
Minimum
1.00
1.00
Median
1.43
1.20
Maximum
1.69
1.30
84
Table 5.39 Variance Estimates for NLSY97 Males: Proportions
Sample Size
4,598
Estimate
(in Percent)
97.40
Design SE
(in Percent)
0.28
SRS SE
(in Percent)
0.23
DEFF
1.22
DEFT
1.10
R in labor force?
4,599
18.59
0.70
0.57
1.23
1.11
R employed?
4,599
18.22
0.68
0.57
1.19
1.09
R working >= 10 hrs/week? (alternative 1)
4,517
9.37
0.50
0.43
1.16
1.08
R working >= 10 hrs/week? (alternative 2)
4,517
12.49
0.60
0.49
1.22
1.11
R employed during school year?
4,599
11.47
0.58
0.47
1.23
1.11
R ever worked during school year?
4,599
21.80
0.76
0.61
1.24
1.12
R employed during summer?
4,599
13.40
0.59
0.50
1.18
1.09
R ever worked during summer?
4,599
17.76
0.65
0.56
1.16
1.08
R ever retained a grade?
3,950
18.11
0.92
0.61
1.51
1.23
R ever skipped a grade?
3,957
1.58
0.23
0.20
1.15
1.07
R ever in remedial classes?
1,907
39.75
1.41
1.12
1.26
1.12
R have a computer at home?
2,791
57.97
1.39
0.93
1.49
1.22
R have a quiet place to study?
2,791
90.68
0.61
0.55
1.11
1.05
R’s parent born in US?
4,080
88.34
1.06
0.50
2.12
1.46
R’s parent has worked at least 3 months since R
born?
DEFF/DEFT Summary
4,074
94.97
0.58
0.34
1.70
1.31
Mean
1.32
1.15
Standard Deviation
0.26
0.11
Minimum
1.11
1.05
Median
1.22
1.11
Maximum
2.12
1.46
Binary Variables
R enrolled?
85
Table 5.40 Variance Estimates for NLSY97 Males: Means
Continuous Variables
Number of weeks worked in 1996?
Sample Size
2,750
Estimated
Mean
9.07
Design SE
0.38
SRS SE
0.30
DEFF
1.27
DEFT
1.12
Days/wk have dinner with family?
2,768
5.35
0.05
0.04
1.25
1.12
Days/wk have fun with family?
2,769
2.71
0.05
0.04
1.25
1.12
Hours/week on homework?
2,407
5.36
0.12
0.09
1.33
1.15
Hours/week watching TV?
2,777
18.31
0.31
0.24
1.29
1.14
Hours/week reading for pleasure?
2,780
2.28
0.10
0.08
1.25
1.12
Parent’s income
2,659
22,633.18
481.41
365.71
1.32
1.15
Spouse’s income
2,003
37,146.39
1,011.58
743.51
1.36
1.17
Percent chance of R getting 4-year degree by age
30
DEFF/DEFT Summary
1,519
66.23
1.02
0.87
1.17
1.08
Mean
1.28
1.13
Standard Deviation
0.06
0.02
Minimum
1.17
1.08
Median
1.27
1.12
Maximum
1.36
1.17
86
Table 5.41 Variance Estimates for NLSY97 Females: Proportions
Sample
Size
4,385
Estimate
(in Percent)
97.50
Design SE
(in Percent)
0.28
SRS SE
(in Percent)
0.24
DEFF
1.17
DEFT
1.08
R in labor force?
4,385
16.33
0.76
0.56
1.36
1.16
R employed?
4,385
16.07
0.75
0.55
1.36
1.17
R working >= 10 hrs/week? (alternative 1)
4,320
8.92
0.53
0.43
1.23
1.11
R working >= 10 hrs/week? (alternative 2)
4,320
11.47
0.61
0.48
1.27
1.13
R employed during school year?
4,385
9.64
0.59
0.44
1.34
1.16
R ever worked during school year?
4,385
17.63
0.76
0.58
1.31
1.14
R employed during summer?
4,385
10.95
0.57
0.47
1.21
1.10
R ever worked during summer?
4,385
14.12
0.65
0.52
1.25
1.12
R ever retained a grade?
3,758
11.75
0.73
0.52
1.40
1.18
R ever skipped a grade?
3,761
2.11
0.27
0.23
1.17
1.08
R ever in remedial classes?
1,965
38.11
1.40
1.10
1.27
1.13
R have a computer at home?
2,619
58.31
1.37
0.96
1.43
1.19
R have a quiet place to study?
2,617
89.56
0.70
0.60
1.17
1.08
R’s parent born in US?
3,856
88.93
0.92
0.50
1.84
1.36
R’s parent has worked at least 3 months since R
born?
DEFF/DEFT Summary
3,853
95.02
0.57
0.35
1.63
1.28
Mean
1.34
1.15
Standard Deviation
0.18
0.08
Minimum
1.17
1.08
Median
1.29
1.14
Maximum
1.84
1.36
Binary Variables
R enrolled?
87
Table 5.42 Variance Estimates for NLSY97 Females: Means
Continuous Variables
Number of weeks worked in 1996?
Sample Size
2,648
Estimated
Mean
7.09
Design SE
0.36
SRS SE
0.28
DEFF
1.28
DEFT
1.13
Days/wk have dinner with family?
2,588
5.03
0.05
0.04
1.25
1.12
Days/wk have fun with family?
2,587
2.63
0.04
0.04
1.00
1.00
Hours/week on homework?
2,316
6.63
0.14
0.11
1.27
1.13
Hours/week watching TV?
2,605
16.17
0.30
0.24
1.25
1.12
Hours/week reading for pleasure?
2,602
3.58
0.11
0.10
1.10
1.05
Parent’s income
2,552
22,309.21
574.34
375.22
1.53
1.24
Spouse’s income
1,794
38,307.95
1,403.80
959.07
1.46
1.21
Percent chance of R getting 4-year degree by age
30
DEFF/DEFT Summary
1,485
73.42
0.96
0.79
1.22
1.10
Mean
1.26
1.12
Standard Deviation
0.16
0.07
Minimum
1.00
1.00
Median
1.25
1.12
Maximum
1.53
1.24
88
Table 5.43 Variance Estimates for NLSY97 Hispanics: Proportions
Sample
Size
1,898
Estimate
(in Percent)
95.68
Design SE
(in Percent)
0.68
SRS SE
(in Percent)
0.47
DEFF
1.45
DEFT
1.20
R in labor force?
1,898
12.33
1.00
0.75
1.33
1.15
R employed?
1,898
11.96
1.01
0.74
1.36
1.17
R working >= 10 hrs/week? (alternative 1)
1,882
6.89
0.75
0.58
1.29
1.14
R working >= 10 hrs/week? (alternative 2)
1,882
8.91
0.86
0.66
1.30
1.14
R employed during school year?
1,898
6.87
0.73
0.58
1.26
1.12
R ever worked during school year?
1,898
14.46
1.05
0.81
1.30
1.14
R employed during summer?
1,898
7.95
0.87
0.62
1.40
1.18
R ever worked during summer?
1,898
11.01
1.04
0.72
1.44
1.20
R ever retained a grade?
1,509
17.72
1.33
0.98
1.36
1.16
R ever skipped a grade?
1,512
1.77
0.41
0.34
1.20
1.10
741
36.00
2.33
1.76
1.32
1.15
R have a computer at home?
1,152
37.76
2.10
1.43
1.47
1.21
R have a quiet place to study?
1,152
87.76
1.26
0.96
1.31
1.14
R’s parent born in US?
1,591
51.66
3.29
1.25
2.63
1.62
R’s parent has worked at least 3 months since R
born?
DEFF/DEFT Summary
1,586
86.82
1.50
0.85
1.76
1.33
Mean
1.45
1.20
Standard Deviation
0.34
0.12
Minimum
1.20
1.10
Median
1.34
1.16
Maximum
2.63
1.62
Binary Variables
R enrolled?
R ever in remedial classes?
89
Table 5.44 Variance Estimates for NLSY97 Hispanics: Means
Continuous Variables
Number of weeks worked in 1996?
Sample Size
1,140
Estimated
Mean
4.79
Design SE
0.49
SRS SE
0.34
DEFF
1.44
DEFT
1.20
Days/wk have dinner with family?
1,149
5.01
0.10
0.07
1.43
1.20
Days/wk have fun with family?
1,146
2.64
0.08
0.06
1.33
1.15
Hours/week on homework?
965
6.63
0.21
0.17
1.24
1.11
Hours/week watching TV?
1,143
18.55
0.57
0.40
1.42
1.19
Hours/week reading for pleasure?
1,143
2.83
0.18
0.14
1.28
1.13
Parent’s income
936
18,629.97
920.59
511.86
1.80
1.34
Spouse’s income
693
25,313.38
1,093.97
750.26
1.46
1.21
Percent chance of R getting 4-year degree by age
30
DEFF/DEFT Summary
573
66.01
1.54
1.34
1.15
1.07
Mean
1.39
1.18
Standard Deviation
0.18
0.08
Minimum
1.15
1.07
Median
1.42
1.19
Maximum
1.80
1.34
90
Table 5.45 Variance Estimates for NLSY97 Non-Hispanic Blacks: Proportions
Sample
Size
2,334
Estimate
(in Percent)
98.06
Design SE
(in Percent)
0.30
SRS SE
(in Percent)
0.28
DEFF
DEFT
1.07
1.04
R in labor force?
2,334
10.40
0.83
0.63
1.32
1.15
R employed?
2,334
10.06
0.81
0.62
1.31
1.14
R working >= 10 hrs/week? (alternative 1)
2,318
6.13
0.61
0.50
1.22
1.10
R working >= 10 hrs/week? (alternative 2)
2,318
7.69
0.66
0.55
1.20
1.10
R employed during school year?
2,334
5.05
0.61
0.45
1.36
1.16
R ever worked during school year?
2,334
12.44
0.93
0.68
1.37
1.17
R employed during summer?
2,334
7.04
0.61
0.53
1.15
1.07
R ever worked during summer?
2,334
10.04
0.81
0.62
1.31
1.14
R ever retained a grade?
1,997
27.22
1.90
1.00
1.90
1.38
R ever skipped a grade?
2,005
2.85
0.49
0.37
1.32
1.15
974
35.00
2.17
1.53
1.42
1.19
R have a computer at home?
1,382
36.01
1.95
1.29
1.51
1.23
R have a quiet place to study?
1,381
87.20
1.07
0.90
1.19
1.09
R’s parent born in US?
2,060
93.14
1.29
0.56
2.30
1.52
R’s parent has worked at least 3 months since R
born?
DEFF/DEFT Summary
2,057
91.13
1.27
0.63
2.02
1.42
Mean
1.44
1.19
Standard Deviation
0.34
0.13
Minimum
1.07
1.04
Median
1.32
1.15
Maximum
2.30
1.52
Binary Variables
R enrolled?
R ever in remedial classes?
91
Table 5.46 Variance Estimates for NLSY97 Non-Hispanic Blacks: Means
Continuous Variables
Number of weeks worked in 1996?
Sample Size
1,428
Estimated
Mean
4.08
Design SE
0.33
SRS SE
0.27
DEFF
1.22
DEFT
1.10
Days/wk have dinner with family?
1,355
4.57
0.09
0.07
1.28
1.13
Days/wk have fun with family?
1,361
2.79
0.08
0.06
1.33
1.15
Hours/week on homework?
1,179
5.81
0.18
0.15
1.20
1.10
Hours/week watching TV?
1,378
23.15
0.53
0.40
1.32
1.15
Hours/week reading for pleasure?
1,379
2.92
0.14
0.13
1.08
1.04
Parent’s income
1,227
20,082.21
868.83
438.59
1.98
1.41
Spouse’s income
583
26,160.00
1,184.22
714.40
1.66
1.29
Percent chance of R getting 4-year degree by age
30
DEFF/DEFT Summary
790
69.76
1.50
1.17
1.28
1.13
Mean
1.37
1.17
Standard Deviation
0.28
0.11
Minimum
1.08
1.04
Median
1.28
1.13
Maximum
1.98
1.41
92
Table 5.47 Variance Estimates for NLSY97 Non-Hispanic Nonblacks: Proportions
Sample
Size
4,731
Estimate
(in Percent)
97.64
Design SE
(in Percent)
0.26
SRS SE
(in Percent)
0.22
DEFF
1.18
DEFT
1.09
R in labor force?
4,732
19.90
0.66
0.58
1.14
1.07
R employed?
4,732
19.60
0.65
0.58
1.12
1.06
R working >= 10 hrs/week? (alternative 1)
4,617
10.22
0.48
0.44
1.09
1.04
R working >= 10 hrs/week? (alternative 2)
4,617
13.46
0.54
0.50
1.08
1.04
R employed during school year?
4,732
12.44
0.59
0.48
1.23
1.11
R ever worked during school year?
4,732
22.34
0.72
0.60
1.20
1.10
R employed during summer?
4,732
14.11
0.56
0.51
1.10
1.05
R ever worked during summer?
4,732
18.20
0.64
0.56
1.14
1.07
R ever retained a grade?
4,195
12.06
0.66
0.50
1.32
1.15
R ever skipped a grade?
4,194
1.63
0.22
0.20
1.10
1.05
R ever in remedial classes?
2,155
40.21
1.43
1.06
1.35
1.16
R have a computer at home?
2,863
66.55
1.36
0.88
1.54
1.24
R have a quiet place to study?
2,862
91.15
0.56
0.53
1.06
1.03
R’s parent born in US?
4,276
94.00
0.59
0.36
1.64
1.28
R’s parent has worked at least 3 months since R
born?
DEFF/DEFT Summary
4,275
97.19
0.46
0.25
1.84
1.36
Mean
1.26
1.12
Standard Deviation
0.23
0.10
Minimum
1.06
1.03
Median
1.16
1.08
Maximum
1.84
1.36
Binary Variables
R enrolled?
93
Table 5.48 Variance Estimates for NLSY97 Non-Hispanic Nonblacks: Means
Sample Size
2,817
Estimated
Mean
9.60
Design SE
0.36
SRS SE
0.31
DEFF
1.16
DEFT
1.08
Days/wk have dinner with family?
2,845
5.36
0.04
0.04
1.00
1.00
Days/wk have fun with family?
2,842
2.65
0.04
0.04
1.00
1.00
Hours/week on homework?
2,569
5.92
0.12
0.10
1.20
1.10
Hours/week watching TV?
2,848
15.75
0.24
0.21
1.14
1.07
Hours/week reading for pleasure?
2,847
2.93
0.11
0.09
1.22
1.10
Parent’s income
3,046
23,463.63
518.66
358.46
1.45
1.20
Spouse’s income
2,516
40,481.38
1,153.45
788.36
1.46
1.21
Percent chance of R getting 4-year degree by age
30
DEFF/DEFT Summary
1,638
70.35
0.92
0.80
1.15
1.07
Mean
1.20
1.09
Standard Deviation
0.16
0.07
Minimum
1.00
1.00
Median
1.16
1.08
Maximum
1.46
1.21
Continuous Variables
Number of weeks worked in 1996?
As shown in the tables above, the variables with the highest design effects throughout the
domains are some of the parent variables, including parent’s and spouse’s income, whether the parent was
born in the U.S., and whether the parent has worked for three or more months since the respondent was
born. Furthermore, for a few variables, means and standard error estimates were based on small sample
sizes. Variables which had sample sizes that were less than one-half of the total NLSY97 sample size
(n=8,984) were whether the respondent has taken remedial classes, spouse’s income, and the percent
chance, according to the parent, that the respondent will earn a four-year college degree by the age of 30.
Finally, Table 5.49 gives a summary of the mean, minimum, and maximum design effects and
root design effects for the six NLSY97 domains. As is typical, the largest means belong to the overall
category. Among the subgroups, the Hispanic and non-Hispanic black means are higher than those of the
other races/ethnicities. Finally, the design effects for males and females are similar.
94
Table 5.49 Comparison of Summary Statistics of NLSY97 DEFTs
Mean
DEFT
Minimum
DEFT
Median
DEFT
Maximum
DEFT
Binary
1.22
1.10
1.18
1.59
Continuous
1.18
1.00
1.20
1.30
Binary
1.15
1.05
1.11
1.46
Continuous
1.13
1.08
1.12
1.17
Binary
1.15
1.08
1.14
1.36
Continuous
1.12
1.00
1.12
1.24
Binary
1.20
1.10
1.16
1.62
Continuous
1.18
1.07
1.19
1.34
Binary
1.19
1.04
1.15
1.52
Continuous
1.17
1.04
1.13
1.41
Binary
1.12
1.03
1.08
1.36
Continuous
1.09
1.00
1.08
1.21
Domain and Types of Variable
Total
Male
Female
Hispanic
Non-Hispanic Black
Non-Hispanic Nonblack
95
Chapter 6. Summary of the Accuracy and Representativeness of the NLSY97 Samples
On the basis of all of the foregoing analyses, we find as follows:
1. The sampling of PSUs and segments was performed in operationally correct fashion, as planned.
2. NORC’s listing operation was apparently successful, because the weighted counts of housing units
listed compare favorably to the Census Bureau’s contemporaneous housing unit counts.
3. The sampling of housing units was performed in operationally correct fashion, as planned.
4. The classification done by NORC field interviewers of housing units as occupied or vacant was
apparently successful, because our vacancy rate compares favorably to the Census Bureau’s
contemporaneous vacancy rates.
5. NORC’s screening of occupied housing units was successful by one standard, achieving over 94
percent response as compared to the 91 percent response rate assumed at the planning stages of
NLSY97. The screener response rate varies from PSU to PSU as one might expect, but four PSUs
ran into computer, and perhaps other, problems and suffered unacceptably low response rates.
6. At the planning stages of NLSY97, we made calculations of the expected number of youth
participants based upon data from the March 1993 Current Population Survey (CPS) and rates from
NLSY79 and other comparable studies. Our calculations were performed in operationally correct
fashion such that, absent undercoverage and differential nonresponse patterns, the expected counts
should reasonably have been realized in the NLSY97 fielding.
7. Nevertheless, the screening interviews failed to find as many eligible youths as expected, but
succeeded in rostering older and younger persons outside the eligible age range. NLSY97 evidently
experienced an undercount specific to the eligible age range.
8. Despite the shortfall of eligible youths, both the screening sample and the samples of NLSY97
participants are reasonably balanced demographically and substantively, as determined by
comparison of the samples to the CPS.
9. For the NLSY97, in comparison to the CPS, we find no major imbalance by sex or by race/ethnicity.
In both the CX and SU samples, there appears to be a relative deficit of Hispanic males. For nonHispanic blacks and for non-Hispanic nonblacks, the NLSY97 sex ratios match the CPS closely.
Within the CX sample, there appears to be a relative surplus of Hispanic and non-Hispanic, black
youths and a relative deficit of non-Hispanic, nonblack youths. This pattern, if true, is particularly
interesting because it is at odds with the well-known pattern of undercount in the U.S. Decennial
Censuses.
10. We do not find a differential undercount by age within the eligible range of 12-16 years. Older
youths age 18-23 found in the screening samples and used in the PAY97 testing program may have
been missed at a higher rate than younger youths age 12-16. However, for the NLSY97, we find no
major imbalance by age.
11. There is no evidence of large substantive bias in the NLSY97 samples, as documented in Appendix F
(PSU-Level Analyses of Socio-Economic-Status Attainment), Appendix G (Comparison of the
NLSY97 Family Income Distribution to the CPS Family Income Distribution), Appendix Q (Analysis
of Parent Education for the NLSY97), and Appendix R (Two Comparisons of Expected vs. Observed
Youth Counts). We note, however, evidence of a modest downward shift in the location of the
96
family income distribution.
12. There is no solid evidence of large coverage differentials in the NLSY97 screening samples by
urban/rural or metro/nonmetro status. There is, however, weak evidence that coverage was more
problematic in urban areas than in rural areas.
13. Coverage of the NLSY97 screening sample compares favorably to the coverage of the CPS for ages
0-11 and 24-35.
14. Youth response rates for the main interview (92 percent overall) exceeded expectation (89 percent
overall).
15. For the NLSY97, we handled nonparticipation through a weighting adjustment, controlling for
sample (CX, SU), race/ethnicity, age, and sex. In our judgement, the adjustment curtailed the
component of nonparticipation bias, if any, associated with the covariates.
16. A review of systems and procedures reveals no technical or substantive errors that would have been
the major cause of either the age hole or the deficit in eligible youths. It is possible that computer
crashes of laptop equipment employed in interviewing caused a minor portion of the deficit.
17. Noninterview reports by field interviewers reveal that screener refusals may be relatively rich in
eligible youths, especially in small HHs. In our judgement, there is some evidence of a differential
response mechanism, whereby HHs with eligible youths had a lower propensity to respond to the
screener than HHs without eligible youths. It is likely that this mechanism contributed to the shortfall
of eligible youths.
18. The advance letter mentions the survey’s focus on youths and the brochure lists the target age range
of interest. Debriefing interviews of the field interviewers show that interviewers frequently revealed
the target group before the completion of the screening interview. All of this information may have
invited reluctant HHs to hide eligible youths or misreport their ages, while apparently cooperating
with the screening interview. It is likely that this mechanism contributed to the shortfall of eligible
youths.
19. The limited evidence we have been able to assemble suggests that households tended to report either
all of their eligible youths or none of them. Partial household misses (reporting some but not all
eligible youths) seem less plausible in the face of our data. Also, it appears the undercount may be
skewed towards single-youth households. A variety of motivations can be ascribed to why the
households may have behaved this way, but of course the screening data and all of our special
analyses of coverage are silent on the intents of the householders.
20. We poststratified the sample by race/ethnicity, age, and sex. In our judgement, the poststratification
curtailed the component of undercoverage bias, if any, associated with the covariates.
21. Survey statistics are appropriately precise. Design effects are consistent across domains, and are
usually well below 1.5. The highest design effects occur for some of the parent variables.
97
22. An independent review of the NLSY97/PAY97 has been conducted by Professor Eugene P. Ericksen
of Temple University19. We have not incorporated his analyses or findings into this report. Among
other things, he examines the possibility that college students were differentially undercounted in the
screening samples, thus contributing to the hole in the age distribution. Even if true, this problem
would have little material effect on the NLSY97 eligible age range of 12-16 years.
23. In the authors’ opinion, taking into account all of the foregoing evidence, the NLSY97 screening
samples missed about 4,000 eligible youths, relative to ideal circumstances. The overall coverage
rate of eligibles was less than 75 percent, while the CPS coverage rate for similar ages is about 85
percent.
24. No natural age distribution known to us exhibits the type of age-specific hole found in the NLSY97
screening sample. It may be reasonable to conclude that the NLSY97 experienced CPS-like
undercoverage of, say, 15 percent, plus additional undercoverage specific to the eligible age range of
about 14 percent (= 28.7 – 15). The additional undercoverage, in our opinion, was caused mainly by
elusive respondents – reacting to information supplied to them by field interviewers and other survey
materials – who refused to report their eligible youths or misreported their ages outside of the eligible
range.
19
Ericksen, E.P. (2000), “An Evaluation of Technical Sampling Procedures of a Profile of American Youth, 1997,”
unpublished manuscript, NORC, Chicago, IL.
98
APPENDIX A
Construction of Demographic Variables
for the NLSY97
99
In order to complete the weighting for NLSY97 as well as provide a basis for other analyses of
the data, complete demographic information was needed. There was a small amount of missing
demographic data for race, ethnicity, sex, and age. Further, there were multiple reports for these variables
which most of the time agreed with one another, but sometimes did not. Thus, the approach put forth in
this appendix was designed to solve two problems. First, if the demographic information was missing, an
imputed value was provided. More importantly, the vast majority of the cases had multiple inputs for
demographic variables; for these, a hierarchy of inputs was established and provided the final
demographic data.
Hispanic Origin
Parent variables:
Parent 1
a) if all orgpar1_1-orgpar1_8 (parent origin) variables are blank, then orgpar1 is blank (missing).
b) if orgpar1_1 is ‘0’ or ‘1’ and all orgpar1_2-orgpar1_8 variables are ‘0’, then orgpar1=’O’ (NonHispanic).
c) if any of orgpar1_2-orgpar1_8 is ‘1’, then orgpar1=’H’ (Hispanic).
The same follows for Parent 2.
On-line questionnaire recodes:
Ethnicity
a) if ethncolq (ethnicity from the OLQ) is blank, then ethnrec (ethnicity recode) is blank (missing).
b) if ethncolq is 1, 2, 3, 4, or 5, then ethnrec=’H’ (Hispanic).
c) if ethncolq is any other value, then ethnrec=’O’ (Non-Hispanic).
Origin
a) if origolq (origin from the OLQ) is blank, then origrec (origin recode) is blank.
b) if origolq is 20, 21, 22, 23, 24, 25, 26, then origrec='H' (Hispanic).
c) if origolq is any other value, then origrec='O' (Non-Hispanic).
Hispanic Origin Variable (FHISP) Construction
1. Screener variable:
• If screener hispanic origin variable (hisposcr) is 0 or 1, FHISP=hisposcr (0=non-Hispanic,
1=Hispanic).
2. OLQ recodes:
• If hisposcr is missing and ethnrec=’H’, then FHISP=1.
• If hisposcr is missing and ethnrec is not ‘H’ but origrec is ’H’, then FHISP=1.
• If hisposcr is missing and ethnrec is ‘O’ and origrec is not ‘H’, then FHISP=0.
• If hisposcr is missing and ethnrec is blank and origrec is ‘O’, then FHISP=0.
3. Parent variable 1:
• If all above variables are missing and orgpar1 is ‘H’, then FHISP=1.
• If all above variables are missing and orgpar1 is ‘O’ or ‘B’, then FHISP=0.
100
4. Parent variable 2:
• If all above variables (including orgpar1) are missing but orgpar2 is ‘H’, then FHISP=1.
• If all above variables (including orgpar1) are missing but orgpar2 is ‘O’ or ‘B’, then FHISP=0.
5. Imputation:
• Cases for which all hispanic origin variables are missing are saved to a file and merged with other
variables in the screening data, which is used for logical imputation. For NLSY97, there were 51
such cases. Those with other family members who are Hispanic were given a value of 1 for
FHISP, and all others were given a value of 0 for FHISP.
Hispanic Origin Source Variable (SHISP) Construction
1. Screener variable:
• If FHISP value is taken from screener variable hisposcr, then SHISP=1.
2. OLQ Ethnicity recode:
• If FHISP value is taken from OLQ variable ethnrec or origrec, then SHISP=2.
3. Parent variable1:
• If FHISP value is taken from orgpar1, then SHISP=3.
4. Parent variable2:
• If FHISP value is taken from orgpar2, then SHISP=4.
5. Imputation:
• If FHISP value is logically imputed with answer file data and given value of 1, then SHISP=5.
• For all remaining cases eligible for imputation but with no available answer file data (FHISP=0),
SHISP=6.
Table A-1 Crosstabulation of Screener Hispanic Origin by OLQ Ethnicity (recoded)
ETHNREC
HISPOSCR
Total
missing
Non-Hispanic
Hispanic
missing
29
29
2062
772
2892
Hispanic
5
5
269
1111
1390
101
NonHispanic
30
18
5383
261
5692
Total
64
52
7714
2144
9974
Table A-2 Crosstabulation of Screener Hispanic Origin by OLQ Origin (recoded)
ORIGREC
HISPOSCR
missing
Non-Hispanic
Hispanic
Total
missing
29
29
2067
773
2898
Hispanic
4
3
114
1037
1158
NonHispanic
31
20
5533
334
5918
Total
64
52
7714
2144
9974
Table A-3 Final Hispanic Origin Frequencies
SHISP
FHISP
0
1
Total
1
2
3
5
6
Frequency
7816
2158
9974
Total
Frequency
9858
58
7
4
47
9974
Sex
Sex Variable (FSEX) Construction
1. Youth variable:
• If youth sex variable (sex_yth) is 1 or 2, FSEX=sex_yth (1=Male, 2=Female).
2. Parent variable:
• If sex_yth is missing and sex_par is 1 or 2, then FSEX =sex_par.
3. Screener variable:
• If sex_yth and sex_par are missing and sex_scr is 1 or 2, then FSEX=sex_scr.
4. Imputation:
• Cases for which all sex variables are missing are saved to a file for imputation. For NLSY97,
there were 84 cases eligible for imputation; 58 of those could be logically imputed based on first
name.
• The remaining 26 cases were randomly imputed (13 male, 13 female).
Sex Source Variable (SSEX) Construction
102
1. Youth variable:
• If FSEX value is taken from sex_yth, then SSEX=1.
2. Parent variable:
• If FSEX value is taken from sex_par, then SSEX=2.
3. Screener variable:
• If FSEX value is taken from sex_scr, then SSEX=3.
4. Imputation:
• If FSEX value is logically imputed, SSEX=4.
• If FSEX value is randomly imputed, SSEX=5.
103
Table A-4 Crosstabulation of Screener Sex by Youth Questionnaire Sex
SEX_SCR
missing
Male
Female
Total
missing
63
21
5056
4789
9929
SEX_YTH
Male
1
2
1
21
25
Female
19
1
20
Total
64
23
5076
4811
9974
Table A-5 Crosstabulation of Screener Sex by Parent Questionnaire Sex
SEX_SCR
missing
Male
Female
Total
missing
64
23
5067
4809
9963
SEX_PAR
Male
Female
9
1
10
1
1
Total
64
23
5076
4811
9974
Table A-6 Final Sex Frequencies
SSEX
FSEX
1
2
Total
Frequency
5121
4853
9974
1
2
3
4
5
Total
104
Frequency
45
6
9839
58
26
9974
Race
Race Variable (FRACE) Construction
1. Screener variable:
• If screener race variable (racescr) is 1, 2, 3, 4, or 5, FRACE=racescr (1=White, 2=Black,
3=American Indian, 4=Asian, and 5=Other).
2. FHISP variable:
• If racescr is missing and FHISP is 1, then FRACE=1.
3. OLQ origin variable:
• If racescr is missing and FHISP=0 but origolq is 1, 3, 5, 6, 8, 9, 10, 13, 16, 17, 20, 21, 22, 23, 24,
25, 26, 27, 28, 29, 30, 31, 32, or 35, then FRACE=1.
• If racescr is missing and FHISP=0 but origolq is 2 or 11, then FRACE=2.
• If racescr is missing and FHISP=0 but origolq is 14, then FRACE=3.
• If racescr is missing and FHISP=0 but origolq is 4, 7, 12, 15, 18, 19, 33 or 34, then FRACE=4.
4. Imputation:
• Cases for which racescr is missing, FHISP=0, and origolq is 36, 37, 88, or missing are saved to a
file and merged with answer file data, which is used for logical imputation. For NLSY97, there
were 58 such cases. 17 of those had other family members with valid race values and were given
that value for FRACE. The remaining 41 did not have any available family race information and
were given a value of 9 for FRACE. Later, for weighting, these 41 cases were folded into the
“Other” (non-Hispanic/non-Black) race category.
Race Source Variable (SRACE) Construction
1. Screener variable:
• If FRACE value is taken from screener variable racescr, then SRACE=1.
2. THISP variable:
• If FRACE value is based solely on FHISP (FRACE=1 for FHISP=1), then SRACE=2.
3. OLQ origin variable:
• If FRACE value is taken from origolq, then SRACE=3.
4. Imputation:
• If FRACE value is imputed, then SRACE=4.
105
Table A-7 Crosstabulation of Screener Race by OLQ Origin (recoded)
RACESCR missing
White
Black
American Indian
Asian
Other
Total
missing
29
79
1473
807
22
55
433
2898
ORIGREC
NonHispanic
Hispanic
Black
7
4
2
22
35
508
1351
64
2
8
1
1
36
551
1434
1158
NonHispanic
Non-Black
24
35
3679
407
38
125
176
4484
Total
64
138
5695
2629
70
182
1196
9974
Table A-8 Final Race Frequencies
FRACE
SRACE
1
2
3
4
5
9
Total
Frequency
5837
2638
76
185
1197
41
9974
1
2
3
4
Total
Frequency
9772
106
38
58
9974
Age
Age Variable (FAGE) Construction
1. Youth date of birth variable:
• If the year portion (first four digits) of the youth date of birth variable (dobyth) is between 1980
and 1984, FAGE = 1996 minus dobyth (dobyth=1980 implies FAGE=16, …, dobyth=1984
implies FAGE=12).
• If the year portion of dobyth is some other year besides 1980-1984 and 1880 (default missing),
FAGE=–1 to indicate age ineligibility at the first stage.
2. Youth age and birthday variables:
• If dobyth is equal to 1880 or missing and ageyth is between 12 and 16, then if bdayyth=0
(indicating that the youth has not had a birthday since January 1, 1997), FAGE=ageyth.
• If dobyth is equal to 1880 or missing and ageyth is between 12 and 16, then if bdayyth=1
(indicating that the youth has had a birthday since January 1, 1997), FAGE=ageyth-1.
106
•
If dobyth is equal to 1880 or missing and ageyth is not missing but is outside of the 12-16 age
range (or if ageyth-1 calculated in the step above is outside of the 12-16 age range), then
FAGE=–2 to indicate age ineligibility at the second stage.
3. Screener date of birth variable:
• If dobyth is 1880 or missing and ageyth is missing, then if the year portion (first four digits) of
the screener date of birth variable (dobscr) is between 1980 and 1984, FAGE = 1996 minus
dobscr (dobscr=1980 implies FAGE=16, …, dobscr=1984 implies FAGE=12).
• If dobyth is 1880 or missing, ageyth is missing, and the year portion of dobscr is some other year
besides 1980-1984 and 1880 (default missing), then FAGE=–3 to indicate ineligibility at the
third stage.
4. Screener age and birthday variables:
• If variables from steps one and two are missing and dobscr is equal to 1880 or missing, then if
agescr is between 12 and 16 and bdayscr=0 (indicating that the youth has not had a birthday
since January 1, 1997), FAGE=agescr.
• If variables from steps one and two are missing and dobscr is equal to 1880 or missing, then if
agescr is between 12 and 16 and bdayscr=1 (indicating that the youth has had a birthday since
January 1, 1997), FAGE=agescr-1.
• If variables from steps one and two are missing and dobscr is equal to 1880 or missing, and if
agescr is not missing but is outside of the 12-16 age range (or if agescr-1 calculated in the step
above is outside of the 12-16 age range), then FAGE=–4 to indicate age ineligibility at the fourth
stage.
5. OLQ year of birth variable:
• If all above variables are missing (or 1880) and yob (year of birth from the on-line questionnaire)
is 80, 81, 82, 83, or 84, then FAGE = 1996-yob.
• If all above variables are missing (or 1880) and yob is not missing and is some other year besides
80, 81, 82, 83, and 84, then FAGE=–5 to indicate ineligibility at the fifth stage.
6. Range variable:
• If all above variables are missing (or 1880) and rangs1ck=1 (indicating that the youth falls into
the 12-16 age range), then FAGE=99 to indicate eligibility for random imputation.
• If all above variables are missing (or 1880) and rangs1ck=0 or missing (indicating that the youth
does not fall into the 12-16 age range), then FAGE=98 to indicate the absence of all age data.
7. Imputation:
• Cases with FAGE=98 or 99 are saved to a file and merged with answer file data, which is used for
logical imputation. For NLSY97, there were 53 such cases. First, all 53 cases were located
within the answer file extract, and if a valid value for age could be found for that person, FAGE
was set equal to that value. 30 cases were given valid FAGE values in this manner. Of the 10
cases given a value of 98 in step 6, only one was not among those 21 cases and was left for
random imputation. Finally, the remaining 22 cases with no age information in the answer files
were given a value of –7, which indicates ineligibility at the last stage. These, along with all
other negative FAGE values, were considered ineligible at the time of weighting.
107
Age Source Variable (SAGE) Construction
1. Youth date of birth variable:
• If a valid FAGE value (12-16) is taken from dobyth, then SAGE=1. Also, if FAGE was deemed
ineligible (FAGE=-1) at the first stage, SAGE=1.
2. Youth age and birthday variables:
• If a valid FAGE value (12-16) is taken from ageyth, then SAGE=2. Also, if FAGE was deemed
ineligible (FAGE=-2) at the second stage, SAGE=2.
3. Screener date of birth variable:
• If a valid FAGE value (12-16) is taken from dobscr, then SAGE=3. Also, if FAGE was deemed
ineligible (FAGE=-3) at the third stage, SAGE=3.
4. Screener age and birthday variables:
• If a valid FAGE value (12-16) is taken from agescr, then SAGE=4. Also, if FAGE was deemed
ineligible (FAGE=-4) at the fourth stage, SAGE=4.
5. OLQ year of birth variable:
• If a valid FAGE value (12-16) is taken from yob, then SAGE=5. Also, if FAGE was deemed
ineligible (FAGE=-5) at the fifth stage, SAGE=5.
6. Range variable:
• For the 10 cases for which rangs1ck=1 and FAGE was given an initial value of 98 and later
imputed, SAGE=6.
7. Imputation:
• Among the remaining 43 cases for which rangs1ck is not equal to 1, 21 were logically imputed
and SAGE set to 7; 22 had no available age data (FAGE=-7) and SAGE was set to 8.
108
Table A-9 Crosstabulation of OLQ Year of Birth by Screener Year of Birth
YOB
missing
00
73
74
75
76
77
78
79
80
81
82
83
84
85
86
88
89
Total
missing
295
3
1957
2
1975
1980
493
4
DOBSCR Year
1981 1982 1983
467
474
466
3
7
10
1
1
1
1984
440
8
1985
4
1991
2
1992
1
4
2
1
1
1
1
1
8
106
96
114
101
94
4
1
1
1
1
5
1195
18
7
1
2
1
18
1396
17
2
1
1
19
1343
18
3
1
2
4
24
1287
22
2
1868
1
1820
2
5
28
1308
25
1
1
824
2
1
1727
1907
1
1818
Total
2644
35
1
2
1
2
2
2
14
1322
1535
1510
1437
1430
32
2
1
2
9974
Table A-10 Crosstabulation of Youth Questionnaire Year of Birth by Screener Year of Birth
DOBYTH
Year
Total
missing
1970
1979
1980
1981
1982
1983
1984
1985
missing
395
2
87
68
99
94
77
2
824
1957
2
2
1975
1980
1608
1
DOBSCR Year
1981 1982 1983
1753 1726 1705
1
95
12
7
2
2
20
124
8
1
1
1
1
17
105
14
4
1
1727
1907
1868
109
1
1
23
82
7
1
1820
1984
1689
1
2
3
28
93
2
1818
1985
4
1991
1992
1
1
1
4
2
1
Total
8883
3
1
207
225
245
221
184
5
9974
Table A-11 Age Frequencies
SAGE
FAGE
Valid
Total
-7
-5
-4
-3
-1
12
13
14
15
16
Frequency
22
3
7
7
9
1936
1984
2041
2049
1916
9974
Valid
Total
110
1
3
4
5
6
7
8
Frequency
1091
8488
196
146
10
21
22
9974
APPENDIX B
This appendix is blank
111
APPENDIX C
This appendix is blank
112
APPENDIX D
Analysis of Language Barrier Cases
113
In this appendix, we analyze the extent of youths designated as Alanguage barrier@ (LB) in
ASVAB testing. We focus mainly on the Enlistment Testing Program, but also present some analysis for
the Student Testing Program. We begin by presenting known national norms for English proficiency.
Later, we compare PAY97 actuals to the national norms and conclude that PAY97 was not unusual in the
extent of LB youths it encountered. In our opinion, this finding also extends to the NLSY97.
National Norms
The 1990 Census long form contained this question:
15a.
Does this person
Yes
No B Skip to 16
speak
a
language
other
than
English
at
home?
15b. What is this language? _________________________
(For example: Chinese, Italian, Spanish, Vietnamese)
15c.
How
well
does
this
person
speak
English?
Very well
Well
Not Well
Not at all.
Using the 1 percent public-use file (U.S. Bureau of the Census, 1993), we cross-tabulated responses to
this question by responses to census question 7, the Hispanic origin question, obtaining Table D-1.
114
Table D-1 Distribution (Percent) of American Youths by Ability to Speak
English and by Hispanic Origin
English Speaking Ability
Not of Hispanic Origin
Hispanic Origin
All Youths
a. Youths Age 12-23
Only English at Home
81.27
92.05
2.95
25.16
84.22
Very Well
4.81
5.44
5.28
45.10
10.09
Well
1.43
1.62
1.68
14.31
3.10
Not Well
0.75
0.84
1.15
9.79
1.89
Not at All
0.04
0.05
0.66
5.64
0.70
Total
88.29
11.71
100.00
b. Youths Age 12-16
Only English at Home
81.62
92.40
3.37
28.90
84.99
Very Well
4.65
5.26
5.62
48.20
10.27
Well
1.31
1.49
1.72
14.77
3.03
Not Well
0.73
0.82
0.73
6.27
1.46
Not at All
0.03
0.04
0.22
1.85
0.25
Total
88.34
11.66
100.00
c. Youths Age 18-23
Only English at Home
81.17
91.95
2.63
22.40
83.8
Very Well
4.82
5.46
4.96
42.35
9.78
Well
1.49
1.69
1.65
14.04
3.14
Not Well
0.75
0.85
1.48
12.67
2.23
Not at All
0.05
0.06
1.00
8.54
1.05
Total
88.28
11.72
100.00
NOTE: Two percentages are given in each cell. The first percent is relative to the population of all youths. The
second percent is relative to the youths in the column domain.
NCES (1994) consolidates persons who responded less than Avery well@ in the category Aspeak
115
English with difficulty.@ Using this consolidation, the census shows 4.75 percent of youths age 12-16
speak English with difficulty. For youths age 18-23, the corresponding figure is 6.42 percent.
Please note that the census question refers to speaking English. The issue of speaking versus
understanding versus writing versus reading is clearly an important one. Some people speak English
passably but can't read it and others read foreign languages reasonably well but speak them abysmally.
Also, the census data are reported either by the person himself or herself or by the census respondent in
the same household.
Macias et al. (1998) report data on limited English proficiency (LEP) reported by State Education
Agencies (SEAs) that receive Title VII funds. We reproduce data for the 1996-97 school year in Table
D-2. SEAs reported a total of 45,650,352 enrolled students, of which 7.4 percent are LEP students.
Table D-2 Summary of Total Student and LEP Enrollments, 1996-97
States and DC
Total Enrollment
LEP Students (in percent)
Public School Students
41,704,542
8.0
Private School Students
3,945,810
0.9
45,650,352
7.4
Total Students
Macias et al. assert that their figures are conservative estimates of the true LEP enrollment, in
part because not every SEA responds to the survey every year and the number of LEP students continues
to increase and become a greater part of the total school enrollment of the country.
Hopstock and Bucaro (1993) provide a careful review and analysis of the strengths and
weaknesses of different methods of estimating the LEP population. In summarizing their own work and
that of other analysts, they suggest that persons who are rated as speaking English less than Avery well@
are likely to be LEP, and that even some rated as speaking Avery well@ may actually be LEP as well. By
this reasoning, census data tell us that at least 5.7 percent of youths age 12-23 should be LEP, while
Macias et al. suggest 7.4 percent of enrolled students are LEP. However, Hopstock and Bucaro conclude
that
What is most clear from all of these differing estimates is that analysts could not agree on criteria
for defining LEP students. Although there was variation based on the data collection
methodologies and population groups included, the most important factor differentiating
estimates appeared to be the standard for determining whether a child was or was not LEP. ... In
order to develop a reliable and consistent count of LEP students, the field will need to agree on an
operational definition of LEP status.
Furthermore, it is far from clear how reporting of LEP status by state agencies compares to the
concept of Alanguage barrier@ as reported by an individual, a household respondent, or a field
interviewer. Estimates from school-based surveys may understate language barrier cases in the general
youth population since students who haven't learned English by high school tend to drop out at a high
rate. Definitions of LEP are far from standard and vary from place to place, and English proficiency is a
116
continuum such that "language barrier" tends to implicate only the more severely limited. For older
youths age 18-23, overall English proficiency will be strongly influenced by immigration of working age
Hispanics and others.
Fleischman and Hopstock (1993) used a mail survey of 745 school districts to estimate the LEP
student population by grade. See Table D-3. According to their data, about 5.5 percent of all students are
thought to be LEP. The numbers are somewhat higher for lower grades than for high school age, e.g., 8.4
percent for kindergarteners, 6 percent for fourth graders, 4.2 percent for eighth graders, and 3.2 percent
for 12th graders.
Table D-3 Number and Percent of LEP Students in Each Grade Level
Grade Level
Total Students in the US
Percent LEP of Total Students
Kindergarten
3,305,619
8.4
1
3,554,274
7.9
2
3,359,193
7.4
3
3,333,285
6.7
4
3,312,443
6.0
5
3,268,381
5.4
6
3,238,095
4.6
7
3,108,120
4.6
8
3,019,826
4.2
9
3,310,290
4.8
10
2,913,951
4.7
11
2,642,554
3.9
12
2,390,329
3.2
---
---
42,000,343
5.5
Ungraded
Total
117
Table D-4 gives a breakdown of the top 10 language groups among LEP students, produced by
the Fleischman and Hopstock survey. Clearly, with 72.9 percent of all LEP students, Spanish is the
dominant non-English language. Other language groups than those shown in Table D-4 individually
account for less than 1 percent of the LEP-student population, and collectively account for only 11.9
percent of the population.
Table D-4 Percentage of LEP Students by Language Group, 1991-92
Language Groups
Percentage of LEP Students
Spanish
72.9
Vietnamese
3.9
Hmong
1.8
Cantonese
1.7
Cambodian
1.6
Korean
1.6
Laotian
1.3
Navajo
1.3
Tagalog
1.1
Russian
0.9
118
Another source of information about LEP youths is the NAEP 1992 mathematics assessments.
Table D-5 provides a summary of some of the results.
Table D-5 Percentages of LEP Students Identified and Excluded from the NAEP Sample
Public Schools
Percent of Students Identified
LEP
Percent of Students Excluded
LEP
a. Grade 4 B 1992
Total US
4
3
Midwest
1
1
Northeast
3
3
South
1
1
West
9
7
22
10
Arizona
9
2
Texas
9
4
Rhode Island
6
3
New York
5
2
California
b. Grade 8 B 1992
Total US
3
2
Midwest
1
1
Northeast
3
2
South
1
1
West
8
4
13
5
Arizona
6
2
Texas
6
2
California
Three percent of eighth graders (13-year-olds) were classified as LEP and 2 percent were
excluded owing to a language barrier. The percent LEP identified and excluded was higher in the West
than in other regions. Several states identified 5 percent or more, including California, Arizona, Texas,
119
Rhode Island, and New York. LEP percentages were a bit higher in the fourth grade than in the eighth
grade assessments.
Generally about a third of students who speak languages other than English in the home have
difficulty speaking English (NCES, 1994, Table 46).
Results from PAY97
Youths included in the PAY97 study were designated as in- or out-of-scope for testing by NORC
field interviewers. Field interviewers performed test scheduling by telephone, and they made their
designation at the time of scheduling. In turn, field interviewers coded youths as to the reason why they
were out of scope for testing, including incapacitated, in jail, language barrier, dead, out of the country, or
in the military. They coded youths as Alanguage barrier@ in the event that the youth could not speak or
understand English on the telephone during the scheduling call, or whenever the field interviewer
otherwise reached the judgment that the youth could not reasonably take the ASVAB test administered in
English.
We present evidence about LB youths in Tables D-6 through D-9. Older youths who screened
into the Enlistment Testing Program (ETP) appear in Tables D-6 and D-7, while younger youths who
screened into the Student Testing Program (STP) appear in Tables D-8 and D-9. About 2.9 percent of all
the older youths (age 18-23) were coded as LB. After removing youths subsampled out, we find that
about 3.1 percent of youths who were age-eligible for the ETP were coded as LB. While more than half
of these LB youths came from the cross-sectional sample, the portion of the supplemental sample (5.3
percent) deemed LB was more than twice as large (2.2 percent) as the corresponding portion of the crosssectional sample. As Table D-7 makes clear, most of the LB youths were of Hispanic origin. Only 7
percent of the LB youths were non-Hispanic. Among Hispanic youths, about 12.4 percent were
designated as LB.
Relatively fewer STP youths were designated LB B only 0.9 percent overall, and 0.8 percent and
1.1 percent for the cross-sectional and supplemental samples, respectively. In absolute terms, the crosssection produced more LB youths, owing to its larger sample size. Over 86 percent of LB youths were
Hispanic, but only 4 percent of Hispanic youths were designated LB.
Because of the larger number (258) of LB youths screened into the ETP, we performed one more
analysis, breaking down the LB cases by geography. We did not perform this analysis for the STP youths
since the sample of LB cases was too small (58). The analysis of ETP youths appears in Figures D-1 to
D3. For each PSU, we calculated the proportion of youths identified as LB and the proportion identified
as Hispanic. We present, in Figure D-1, a scatter plot of proportion LB versus proportion Hispanic across
the total sample of 200 PSUs. Figures D-2 and D-3 contain similar scatter plots for the 100 crosssectional and 100 supplemental PSUs, respectively. Most of the PSUs have a very low proportion of LB
cases. On the other end, many of the PSUs with a high proportion LB also have a high proportion
Hispanic. This finding is not surprising and, in fact, is intimated by earlier tables. Three cross-sectional
PSUs and 14 supplemental PSUs have a proportion LB greater than 0.1, and we list these PSUs in Table
D-10. Of these 17 PSUs, however, 11 are based upon a very small numbers of cases B probably too small
to warrant attention. Clearly, most of the LB cases come from densely Hispanic areas and from areas of
the country known for their Hispanic populations.
120
Table D-6 ETP Status * Sample Crosstabulation
ETP Status
Ineligible, subsampled out
Age ineligible or special
circumstances
OOS, incapacitated
OOS, in jail
OOS, language barrier
OOS, dead
OOS, out of country
OOS, in military
Eligible, nonparticipant
Eligible, incomplete test
Eligible, AFQT complete
Eligible, complete test
Total
Count
% within NEWCODE
% within Sample
% of Total
Count
% within NEWCODE
% within Sample
% of Total
Count
% within NEWCODE
% within Sample
% of Total
Count
% within NEWCODE
% within Sample
% of Total
Count
% within NEWCODE
% within Sample
% of Total
Count
% within NEWCODE
% within Sample
% of Total
Count
% within NEWCODE
% within Sample
% of Total
Count
% within NEWCODE
% within Sample
% of Total
Count
% within NEWCODE
% within Sample
% of Total
Count
% within NEWCODE
% within Sample
% of Total
Count
% within NEWCODE
% within Sample
% of Total
Count
% within NEWCODE
% within Sample
% of Total
Count
% within NEWCODE
% within Sample
% of Total
121
Sample
Cross-Sectional
Sample
593
97.4%
8.7%
6.7%
38
54.3%
.6%
.4%
20
66.7%
.3%
.2%
9
39.1%
.1%
.1%
147
57.0%
2.2%
1.7%
48
63.2%
.7%
.5%
31
81.6%
.5%
.3%
7
70.0%
.1%
.1%
1266
72.2%
18.6%
14.2%
17
45.9%
.2%
.2%
79
77.5%
1.2%
.9%
4566
77.4%
66.9%
51.3%
6821
76.6%
100.0%
76.6%
Supplemental
Sample
16
2.6%
.8%
.2%
32
Total
609
100.0%
6.8%
6.8%
70
45.7%
1.5%
.4%
10
33.3%
.5%
.1%
14
60.9%
.7%
.2%
111
43.0%
5.3%
1.2%
28
36.8%
1.3%
.3%
7
18.4%
.3%
.1%
3
30.0%
.1%
.0%
488
27.8%
23.4%
5.5%
20
54.1%
1.0%
.2%
23
22.5%
1.1%
.3%
1334
22.6%
64.0%
15.0%
2086
23.4%
100.0%
23.4%
100.0%
.8%
.8%
30
100.0%
.3%
.3%
23
100.0%
.3%
.3%
258
100.0%
2.9%
2.9%
76
100.0%
.9%
.9%
38
100.0%
.4%
.4%
10
100.0%
.1%
.1%
1754
100.0%
19.7%
19.7%
37
100.0%
.4%
.4%
102
100.0%
1.1%
1.1%
5900
100.0%
66.2%
66.2%
8907
100.0%
100.0%
100.0%
Table D-7 ETP: OOS, Language Barrier * Hispanic Origin Crosstabulation
OOS, Language Barrier
No
Yes
Total
Count
% within OOS, language barrier
% within Hispanic Origin
% of Total
Count
% within OOS, language barrier
% within Hispanic Origin
% of Total
Count
% within OOS, language barrier
% within Hispanic Origin
% of Total
122
Hispanic Origin
NonHispanic Hispanic
6938
1698
80.3%
19.7%
99.7%
87.6%
78.0%
19.1%
18
240
7.0%
93.0%
.3%
12.4%
.2%
2.7%
6956
1938
78.2%
21.8%
100.0% 100.0%
78.2%
21.8%
Total
8636
100.0%
97.1%
97.1%
258
100.0%
2.9%
2.9%
8894
100.0%
100.0%
100.0%
Table D-8 STP Status * Sample Crosstabulation
Sample
Cross-Sectional Supplemental Sample
Sample
14
19
42.4%
57.6%
.3%
.9%
.2%
.3%
11
8
57.9%
42.1%
.3%
.4%
.2%
.1%
11
3
78.6%
21.4%
.3%
.1%
.2%
.0%
35
23
60.3%
39.7%
.8%
1.1%
.5%
.4%
50
26
65.8%
34.2%
1.2%
1.2%
.8%
.4%
3
2
60.0%
40.0%
.1%
.1%
.0%
.0%
1
100.0%
.0%
.0%
903
569
61.3%
38.7%
21.4%
26.4%
14.2%
8.9%
19
25
43.2%
56.8%
.4%
1.2%
.3%
.4%
42
10
80.8%
19.2%
1.0%
.5%
.7%
.2%
3135
1467
68.1%
31.9%
74.2%
68.2%
49.2%
23.0%
4224
2152
66.2%
33.8%
100.0%
100.0%
66.2%
33.8%
STP Status
Age or grade ineligible, or Count
special circumstance
% within STP Status
% within Sample
% of Total
OOS, incapacitated
Count
% within STP Status
% within Sample
% of Total
OOS, in jail
Count
% within STP Status
% within Sample
% of Total
OOS, language barrier
Count
% within STP Status
% within Sample
% of Total
OOS, dead
Count
% within STP Status
% within Sample
% of Total
OOS, out of the country
Count
% within STP Status
% within Sample
% of Total
OOS, in military
Count
% within STP Status
% within Sample
% of Total
Eligible, nonparticipant
Count
% within STP Status
% within Sample
% of Total
Eligible, incomplete test Count
% within STP Status
% within Sample
% of Total
Eligible, AFQT complete Count
% within STP Status
% within Sample
% of Total
Eligible, complete test
Count
% within STP Status
% within Sample
% of Total
Total
Count
% within STP Status
% within Sample
% of Total
123
Total
33
100.0%
.5%
.5%
19
100.0%
.3%
.3%
14
100.0%
.2%
.2%
58
100.0%
.9%
.9%
76
100.0%
1.2%
1.2%
5
100.0%
.1%
.1%
1
100.0%
.0%
.0%
1472
100.0%
23.1%
23.1%
44
100.0%
.7%
.7%
52
100.0%
.8%
.8%
4602
100.0%
72.2%
72.2%
6376
100.0%
100.0%
100.0%
Table D-9 STP: OOS, Language Barrier * Hispanic Origin Crosstabulation
OOS, Language Barrier No
Yes
Total
Count
% within OOS, Language Barrier
% within Hispanic Origin
% of Total
Count
% within OOS, Language Barrier
% within Hispanic Origin
% of Total
Count
% within OOS, Language Barrier
% within Hispanic Origin
% of Total
124
Hispanic Origin
NonHispanic
Hispanic
5114
1204
80.9%
19.1%
99.8%
96.0%
80.2%
18.9%
8
50
13.8%
86.2%
.2%
4.0%
.1%
.8%
5122
1254
80.3%
19.7%
100.0%
100.0%
80.3%
19.7%
Total
6318
100.0%
99.1%
99.1%
58
100.0%
.9%
.9%
6376
100.0%
100.0%
100.0%
Table D-10 Youths Screened into the ETP: PSUs with Proportion
Language Barrier Greater than or Equal to 0.10
PSU Number
Sample
Number of LB
Youths
Proportion LB
Proportion
Hispanic
704
Supplemental
3
0.14
0.59
721
Supplemental
3
0.20
1.00
723
Supplemental
2
0.12
0.29
747
Supplemental
20
0.44
0.89
761
Supplemental
22
0.13
0.91
766
Supplemental
17
0.36
0.83
785
Supplemental
4
0.20
1.00
792
Cross-Sectional
57
0.18
0.58
795
Supplemental
2
0.25
0.25
812
Cross-Sectional
13
0.19
0.25
825
Supplemental
1
0.13
1.00
835
Supplemental
2
0.12
0.53
841
Supplemental
11
0.11
0.59
863
Supplemental
2
0.18
1.00
868
Cross-Sectional
4
0.12
0.70
872
Supplemental
2
0.13
1.00
896
Supplemental
1
0.50
1.00
Findings
As we have remarked previously, language proficiency is a difficult concept to define and
measure well. Even in this brief appendix, we have introduced three specific terms B LB, LEP, and speak
English with difficulty B which are related to language proficiency, and it is unclear exactly how these
terms relate definitionally to one another. We suppose, however, that LB could represent the most
seriously English deficient level.
128
Even if the terminology were well defined and understood, different sources of data arise from
different measurement processes. Some data are self-reported by the person. Some data are reported by a
household informant; others by an interviewer; and still others by a school or school district.
Comparability of data sources is also compromised by the fact that different sources cite data for
different years. Some sources are not even internally consistent as to reference period.
We conclude, without question, that the data on English proficiency displayed in this appendix
are subject to a variety of differential biases and random errors.
Despite these limitations, we summarize the data in Table D-11. From these data, there is no
evidence that the ETP or STP samples were excessively saturated with LB youths. Quite to the contrary,
these data suggest that the PAY97 samples found LB youths at a typical to below typical rate.
Furthermore, census data suggest that there should be more LBs among youths age 18-23 than among
youths 12-16, and this general pattern is present in the ETP and STP data.
Table D-11 Estimates of English Language Deficiency
Source
Percent English Deficient
STP, LB Youths, Grades 10-12
0.9
1990 Census, ANot Well@ plus ANot at All@, Age 12-16
1.7
NAEP, LEP Students, Grade 8
3.0
ETP, LB Youths, Age 18-23
3.1
Fleischman and Hopstock, LEP Students, Grade 12
3.2
1990 Census, ANot Well@ plus ANot at All@, Age 18-23
3.3
Fleischman and Hopstock, LEP Students, Grade 11
3.9
NAEP, LEP Students, Grade 4
4.0
Fleischman and Hopstock, LEP Students, Grade 10
4.7
1990 Census, ASpeak English with Difficulty@, Age 12-16
4.8
Fleischman and Hopstock, LEP Students, Age 5-17
5.5
1990 Census, ASpeak English with Difficulty@, Age 18-23
6.4
Macias et al., LEP Students
7.4
National norms further suggest that the majority of English language deficient cases should be
Hispanic, and that the West, and California in particular, should display higher rates of deficiency than
other geographic areas. These patterns too are present in ETP and STP data. For example, Fleischman
and Hopstock=s data suggest that around 73 percent of LEP students are Hispanic. On the other hand, the
ETP finds that 93 percent of LBs are Hispanic. That ETP finds more Hispanics than Fleischman and
Hopstock is not surprising because Hispanics were oversampled for the ETP.
129
Among Hispanic youths age 18-23, the census reports that 12.7 percent speak English Anot well@
and another 8.5 percent speak Anot at all.@ By comparison, in the ETP, 12.4 percent of Hispanic youths
were designated LB. Among Hispanic youths age 12-16, the census reports 0.7 percent speak Anot well@
and another 0.2 percent speak Anot at all.@ For STP youth, 4.0 percent of Hispanics were designated LB.
The differences between census and PAY97 data are not large, and in any event are confounded by
definitional issues, reporting issues, and the PAY97 oversample.
In light of the foregoing evidence, we find no support for the claim that the PAY97 samples are
excessively saturated with LB cases. We find no support for the claim that LBs compromise the
representativeness of the PAY97 samples to a greater extent than they compromise the representativeness
of other important, national surveys.
References
Fleischman, H.L. & Hopstock, P.J. (1993), Descriptive Study of Services to Limited English Proficient
Students. Development Associates, Inc., Arlington, VA.
Hopstock, P.J., and Bucaro, B.J. (1993), A Review And Analysis of Estimates of the LEP Student
Population. Development Associates, Inc., Arlington, VA.
Macias, R.F., et al. (1998), Summary Report of the Survey of the States’ Limited English Proficiency
Students and Available Educational Programs and Services, 1996-97. Washington, DC: National
Clearinghouse for Bilingual Education.
Mullis, I.V.S., et al. (1993), NAEP 1992 Mathematics Report Card for the Nation and the States.
National Center for Education Statistics, Washington, D.C.
National Center for Education Statistics (1994), The Condition of Education, 1994. National Center for
Education Statistics, Washington, D.C.
U.S. Bureau of the Census (1993), Census of Population and Housing, 1990: Public Use Microdata, 1
Percent Sample. CD-ROM, 2 disks, 2nd release, distributed by the Bureau of the Census, Washington,
D.C.
130
APPENDIX E
Review of Proxy Interviews
131
Introduction
This report summarizes the results of reviewing a statistical sample of the record of calls (ROCs)
for cases given a field disposition of 74, proxy screener, by NORC field interviewers. There were 5,175
households identified as “proxy” during the data collection and 490 were reviewed. The purpose of the
review was to ensure that the proper procedures were followed and that there were no indications of
youths in the households due to improper use of this disposition code.
During the preset of Round 1, NORC and BLS realized that many households were extremely
hard to reach. Field interviewers were making ten to twenty visits and still finding no one home. NORC
and BLS needed to find more cost-efficient ways to determine if there were eligible youths in households.
After the Round 1 pretest, NORC and BLS decided to allow the field interviewers to receive help from
specified neighbors in determining whether youths lived at the sample households. While there was
worry about misuse of this procedure, this review shows that there were few cases where errors were
found.
Rule for Using a Proxy Screener
The field interviewer must have had no contact with anyone in the selected housing unit and have
attempted to contact the household at least three times (one daytime attempt not on the weekend, one
evening attempt not on a Friday, Saturday or Sunday, and one weekend attempt), and then the field
interviewer was allowed to contact a neighbor to determine if the housing unit was occupied, if the
residents were on vacation, whether they were just hard to find, and when it would be a good time to find
them at home. In addition, the field interviewers were to attempt to determine the household’s
composition from the neighbor.
Basically, the purpose of talking to the neighbor and administering the proxy screener was to
assess whether there were eligible members residing in the sampled household and to determine what
steps were needed to complete the screener roster.
In order for field interviewers to complete the CAPI screener with a proxy they had to make sure
that the proxy was a responsible adult, lived either next door to or directly across the street (or hallway)
from the selected housing unit, was aware of the exact number of persons living in the selected housing
unit, and were positive that there were no persons between the ages of 8 and 28 living in the selected
housing unit. If the field interviewers were unable to locate a proxy who met all of these criteria, they
had to continue attempting to reach and screen the housing unit.
If the field interviewer learned from the proxy screener that there was no one in the eligible age
range in the sample household, they had to make at least three more attempts to contact and screen the
sample household. If they still could not make contact with anyone in the household, they had to consult
with their field managers before finalizing the case based on the proxy data and assigning a proxy code.
However, if the proxy household appeared to contain someone within the specified age range, or
the neighbor did not know for sure if there were youths in the household, the field interviewer was to
continue his or her efforts to contact the sampled household based on the suggestion of the nieghbor as to
the best time to reach the household members. If they were still unsuccessful after having done all they
could to contact the household, they were to consult with their field manager to determine if they should
end their attempts to contact this household. If the field manager determined the case should be finalized,
the case would have been assigned a final noninterview code, but would not have been given a proxy
code.
Records of Calls
132
NORC uses a Case Management System (CMS) to control the disposition of cases. In the CMS,
the field interviewers have the ability to enter information about efforts to get an interview. They are not
required to enter a record of call (ROC) for every attempt, phone call or visit. For example, they may go
by a house several times in one day, but only enter one ROC. Therefore ROCs can’t be used to determine
number of contacts. The field interviewers were not told to enter the number of household members or
whether there were youths. However, many field interviewers did in fact enter information that some
number of youths were in the household, even when they were unable to get a completed interview. They
tended to make statements like, “I saw teenagers coming out of the house” or that a neighbor said there
were eligible youths in the household. The fact that they did not enter such information does not mean
they made a mistake since there was no requirement to do so, just as there was no request to enter a ROC
for every contact with the household. It was thought to be too time consuming to enter a ROC for every
visit, since they often stop by a sample address several times a day when they are in the area. Opening
and closing the case in a laptop computer takes valuable time. In future screenings we would recommend
that they be required to enter any possible information about household members in a note to the ROC
and to enter a ROC for every household contact. This information is very useful when reviewing
information about the sample.
Sample
The 5,175 proxy screeners were sorted by field interviewers with increasing numbers of proxy
cases. Five samples of approximately 98 cases each were drawn. Starting with case seven, using a
selection interval of 53, the first sample was selected. This process was repeated until five independent
samples were drawn. The purpose of drawing five samples was to decrease the work of the person
reviewing the ROCs. If no problems were found in the first sample, then review of the other samples
might not be necessary. However, the reviewer decided to review all five samples to be certain there
were no problems. As a result, 490 screener ROCs were reviewed.
Results of Review
A review of the ROCs for a statistical sample of 490 households given a code 74, proxy screener,
found that one household had three youths in the 17- to 23-year-old age range (ASVAB-eligible). Two
households were found that had one or more eligible youths in the 12- to 16-year-old age range
(NLSY97-eligible). If you extrapolate to the total number of 5,175 proxy screeners, one would find only
11 ASVAB-eligible households and 21 NLSY97-eligible households. Approximately 77 percent of the
sample had specific notes in either the ROCs or the Noninterview Report that there were no eligible
youths in the household and slightly over 22 percent had no such statement, but there were also no notes
indicating youths were present. Since such a note was not required, one should assume these were done
correctly and there are no youths. Previous reviews of ROCs have indicated that field interviewers are
much more likely to have written a note when there were youths than when there were not.
Slightly over three percent of the time, the field interviewer gave a disposition of proxy when the
respondent told her/him that there were no eligible youths. These cases should have been coded as
refusals. If one extrapolates to the full sample, there would have been approximately 169 such cases.
Likewise, field interviewers accepted statements from housekeepers and babysitters about one percent of
the time. These should not have been proxy households.
133
Approximately two percent of the sampled cases should have been codes as vacant, demolished
or some other Type B or C noninterview.
Most of the time (92.2 percent), field interviewers correctly followed the procedure. In those
cases where procedures were not followed, only 0.6 percent were households with possible youths
present.
Table E-1 Review of Proxy Record of Calls
Number of
Households in
Sample
Percent of
Households in
Sample
Total#
490
100
No kids, definitive statement
No definitive statement, but no kids indicated
Respondent said no one eligible****
Housekeeper, babysitter said no one eligible
No kids, crash
No kids, should be vacant
No kids, should be Type C (demolished, etc)
No kids, bad address
Possible age 17-23 in household
Possible age 12-16 in household
343
109
16
5
4
8
2
0
1
2
70.00
22.24
3.27
1.02
0.82
1.63
0.41
0.00
0.20
0.41
Description*
*Interviewers were not instructed to write any particular type of note in the record of
calls, so the fact that they did not indicate youths does not mean they erred.
****Technically these should have been refusals.
#
Totals may not add due to rounding.
134
APPENDIX F
PSU-level Socio-Economic Status Analysis
135
In this appendix, we discuss how PSU-level screener response and other rates depend on socioeconomic status, as represented in this report by education and income. For each category, we actually
use two different variables (although they are correlated). For education, we use two summary variables:
percent of PSU adults who are at least high school graduates, and percent of PSU adults who have at least
a bachelor’s degree. It is important to note that these percentages, taken from the 1990 Census, include
all persons, male and female, who are at least 25 years of age. Therefore, these percentages are likely to
be smaller than the percentages would be for just NLSY97/PAY97 parents (e.g., older generations went to
college less, especially females).
For income, we use the median family income for the PSU and the median household income for
the PSU. Family income is larger (which seems counter-intuitive if families are subsets of households)
because single, unrelated persons would be considered separate households, not families (families consist
of two or more related persons).
The PSU-level NLSY97/PAY97 rates studied are the screener response rate, the percentage of
gatekeeper cases, the percentage of out-of-scope cases, and the percentage of proxy cases. These four
variables will often be referred to as the dependent variables. The income and education variables will
often be referred to as the independent variables.
Education
As stated above, the two education variables studied are percent of high school graduates and
percent of bachelor degree recipients. The correlation between the two percentages is .727, but the
bachelor degree recipients variable seems to have a stronger relationship with the dependent variables, as
shown by the correlation matrix below:
Table F-1 PSU-level Socio-Economic Correlations: Education
Percent High
School Graduate
or Higher
Pearson Correlation
Percent Bachelors
Degree or Higher
Pearson Correlation
PSU
Screener
Response
Rate
PSU
Screener
Gatekeeper
Rate
PSU
Screener
Proxy
Rate
PSU
Screener
Out-ofScope Rate
-.065
.058
.084
-.331
.361
.412
.237
.000
-.164
.190
.139
-.358
.021
.007
.050
.000
Significance (2-tailed)
Significance (2-tailed)
136
Table F-1 shows that the correlations with the independent variables are in the same direction for
both dependent variables, but that the relationships are much stronger with the percent bachelor=s degree
variable. The basic relationships seem to suggest that more educated PSUs are less likely to respond,
more likely to have gatekeeper communities, more likely to need proxy information, and less likely to
have out-of-scope cases. For PSUs with more high school graduates, these relationships are quite weak,
except for the out-of-scope rate. This suggests that more educated people live in residential areas (e.g.,
there are few businesses, group quarters, etc.), and that these residential areas probably have fewer
condemned, demolished, and vacant housing units.
With regard to the bachelor=s degree variable, all four correlations are significant, although the
correlation with the PSU proxy rate is only marginally significant (p=.050). Of particular interest, of
course, is the relationship with the PSU screener response rate. Here is a scatterplot of PSU screener
response rates by PSU college graduate percentages:
Figure F-1 Scatterplot of PSU Screener Response Rate by PSU Bachelor Degree
(or Higher) Rates
It should be noted that the outlier PSUs with less than an 80 percent screener response rate are omitted
from the above graph so that there can be more differentiation among the other data points. A downward
pattern seems clear, but the outline shape of the data overstates the relationship because of the increasing
variability among PSU screener response rates as the percentage of bachelor degree recipients rises. A
simple unweighted regression (line shown in graph) gives the following equation:
Y = 96.9 - .09 X
where Y= Screener Response Rate (percentage)
and
X= Percentage of Bachelor Degree holders
137
This equation implies that a PSU with 10 percent bachelor degree recipients would have a 96.0 percent
response rate, as compared to a 94.2 percent response rate for a PSU with 30 percent bachelor degree
recipients.
Income
As stated above, the two response variables studied are two different measures of PSU median
income: median family income and median household income. A family is defined as two or more
related persons living together. Unrelated single persons living together are considered to be different
households, and the housing unit is considered to have zero families. Nevertheless, the median incomes
by family or by household are highly correlated. The correlation between the two income variables is
.987, so there is very little difference in their relationships with the independent variables, as shown by
the correlations below.
Table F-2 PSU-level Socio-Economic Correlations: Income
Median
Household
Income, 1990
Census
Pearson Correlation
Median Family
Income, 1990
Census
Pearson Correlation
Significance (2-tailed)
Significance (2-tailed)
PSU
Screener
Response
Rate
PSU
Screener
Gatekeeper
Rate
PSU
Screener
Proxy Rate
PSU
Screener
Out-ofScope Rate
-.105
.137
.193
-.404
.140
.054
.006
.000
-.103
.126
.198
-.406
.148
.075
.005
.000
It can be seen that as income rises, there is a slight drop in the screener response rate, but not a significant
drop. However, income relates more directly to the proxy and out-of-scope rates than the education
variables do. While the p-values for the education variable correlations with the proxy rate were not
significant or marginally significant, the p-values in Table F-2 for the proxy rate are very significant.
Proxies were needed more often in higher-income areas. Also, the correlation between the out-of-scope
rate and income is -.40, as compared to a correlation of -.35 between the out-of-scope rate and education.
Just as more educated persons are more likely to live in Astable@ residential areas, these numbers suggest
that high income persons are even more likely.
138
Figure F-2 is a scatterplot of the PSU screener response rate with median family income.
Figure F-2 Scatterplot of PSU Screener Response Rate by
PSU Median Family Income
Once again, the outlier PSUs with less than an 80 percent screener response rate are omitted from the
above graph, and the outline shape of the data overstates the downward trend of the data, as shown by the
regression line. The simple regression line corresponds to the following equation:
Y = 97.3 - .6 X
where Y= Screener Response Rate (percentage)
and
X= Median Family Income (in $10000s)
This equation implies that a PSU with a median family income of $20,000 would have a 96.1 percent
response rate, as compared to a 94.3 percent response rate for a PSU with a median family income of
$50,000.
139
Summary
Differences in socio-economic status do lead to differences in certain screener rates. In particular,
PSUs with a higher socio-economic status have many fewer out-of-scope cases. Out-of-scope cases
include condemned and demolished buildings, businesses, unlocatable addresses, group quarters, vacation
cabins and other non-permanent residences, and transient and vacant buildings. It is not surprising that
PSUs with a higher socio-economic status seem to have more of a purely residential mix. In fact, the
linear regression equation involving the median family income and the out-of-scope rate is:
= 25.9 - 3.96 X
where Y= PSU Out-of-Scope Rate (percentage)
and
X= Median Family Income (in $10000s)
This equation implies that a PSU with a median family income of $20,000 would have an 18.0 percent
out-of-scope rate, as compared to a 6.1 percent out-of-scope rate for a PSU with a median family income
of $50,000. This is a much larger difference than we saw above for the PSU screener response rates.
This large difference, of course, does not suggest an NLSY97/PAY97 bias, since out-of-scope cases are
just that . . . out-of-scope, and eliminated from all analyses.
PSUs with a higher socio-economic status did have higher proxy and gatekeeper rates on both
education and income. This means they were harder to find at home, but these two codes were only to be
used when the information provided was that there were no persons between 8 and 28 years old in the
housing unit. The eight correlations (two rates by four socio-economic variables) all range between 0 and
0.2. Neither the proxy nor gatekeeper rates are significantly correlated with percent of high school
graduates. However, the gatekeeper rate is significantly correlated with percent of bachelor degree
recipients, and the proxy rate correlation is marginally significant. For the income variables, the
gatekeeper correlations are almost significant (p=.054 for household income and p=.075 for family
income), but the proxy correlations are quite significant (both p < .01). These differentials in the proxy
and gatekeeper rates could lead to bias if significant numbers of eligible youths were missed within proxy
and gatekeeper cases. However, NORC examined 490 proxy cases and 494 gatekeeper cases. Only three
Apossible@ eligibles were discovered in each sample, making it likely that any possible bias (or
contribution to the Amissing@ youths problem) due to missed eligibles in proxy and gatekeeper cases is
very small.
For the most important dependent variable studied, there was a decrease in the screener response
rate in PSUs with higher socio-economic status. However, the only screener response rate correlation that
was significant was with percent of bachelor degree recipients. In both education and income, the
difference in screener response rates was about 96 percent for low-SES PSUs, and 94 percent for highSES PSUs. This 2 percent difference is about the same as differences seen between metropolitan and
non-metropolitan areas, and is not much larger than the 1.5 percent difference seen among the four
Census regions. Therefore, there does not appear to be a serious bias here, although there is certainly a
slight bias in that it was more difficult to collect screener data from high-SES households. Also, as a final
caution, it should be noted that the correlations could be muted by the fact that we only have PSU-level
data, as opposed to non-response rates for individual households that have different socio-economic
statuses.
140
APPENDIX G
Comparison of the NLSY97 Family
Income Distribution to the CPS Family
Income Distribution
141
In order to further explore the possible differences between NLSY97 youths and the general
population of 12- to 16-year-olds, NORC conducted an analysis based on family income. Income was
examined according to four classifications: (1) wages and salary income, (2) self-employment income
(which includes both non-farm and farm income), (3) total earned income (the sum of wages and salary
and self-employment income), and (4) total income from all sources. The frequencies of NLSY97
families within these income classifications were tabulated overall, for the cross-sectional sample, and by
race/ethnicity. For comparison, the same frequencies were run for families with resident youths aged 12
to 16 responding to the March 1997 Current Population Survey (CPS), overall and by race/ethnicity.
Framework for Income Analyses. The analyses that follow were conducted at the youth level,
and were based on family, rather than household, income. NLSY97 data provided detailed income
information for the youths’ parents and parents’ spouses, but not necessarily for all household members.
These family-level income data matched most closely to the CPS family income; thus, the family was
chosen as the basis for the income analysis. However, rather than calculating income distributions across
families, we viewed family income as a youth characteristic, and then calculated weighted distributions
across all youths. For the NLSY97, family income was calculated based on the income of the parent or
parents living with the youth. For the CPS, all 12- to 16-year-old youths were selected for analysis, and
family-level income was calculated for each.
In the case of the CPS, final person weights were used for computing income distributions. For
the NLSY97, we used the final youth weights, which combine the cross-sectional and supplemental
samples. A separate weight was used when analyzing the cross-sectional sample alone; this crosssectional weight reapportions the weights across only the cross-sectional sample, giving all supplemental
cases a weight of zero. Note that comparisons have not been made between the supplemental sample and
the CPS; because the supplemental sample is made up only of Hispanic and non-Hispanic black youths, it
would be unfair to compare their families to the families of all CPS youths. A comparison based on
race/ethnicity will reveal any differences between NLSY97 and CPS minority and non-minority youths.
Both the CPS and NLSY97 surveys ask similar income questions. The CPS asks two questions
about each type of income for which it collects data. First it asks whether any income was received; then
it asks for the amount (value) of income received. The NLSY97 also asks these two questions;
furthermore, if the respondent refuses to give the exact income amount, it asks them to select, from a list
of ranges, one into which their income falls. Due to this variation, the construction of the income
variables for the two surveys entailed slightly different approaches. As mentioned above, the four
household income variables constructed for comparison purposes were wages and salary income, selfemployment income, total earned income, and total income from all sources. For each income variable in
the CPS, the given income value was used as the raw income, and an additional variable was created to
group this value into the seven categories used in the tables in this analysis: $1-$5,000; $5,001-$10,000;
$10,001-$25,000; $25,001-$50,000; $50,001-$100,000; $100,001-$250,000; and more than $250,000.
For the NLSY97, if the income value was given, it was used as the raw income. If instead a range was
given, the midpoint of the range was used as the raw income value. (For example, if the respondent
selected the range $5,001-$10,000 for a particular income type, they were given a raw value of $7,500 for
that income type.) Finally, an additional variable was created to collapse raw income into the same seven
analysis categories described for the CPS above.
The raw income values discussed above may, in some cases, be topcoded. Because the NLSY97
data come from the public use file, the values above a certain threshold are subject to a topcoding scheme,
devised by CHRR, whereby the top two percent of the income values were truncated. Those values
which exceeded the truncation level were averaged, and each of those cases was given the average value.
We are also aware that CPS values above a particular threshold are topcoded; however, the method by
which this is accomplished is complicated. The March 1997 CPS documentation contains an explanation
of their topcoding system.
142
In this analysis, up to two forms of null data were coded as missing. For the first three income
classifications, families which refused to give an income value (or range) had their income coded as
missing; all other values were considered valid. For total income, not only were refusals coded as
missing, but so too were zero-income values (i.e., families reporting having received no income
whatsoever). In other words, families that either refused to disclose income amount or that claimed to
have no income from any income source were given a missing status for that variable. Because both the
CPS and the NLSY97 surveys ask numerous questions about types of income received, and both have a
catchall “all other income” question, it is highly unlikely that a family would have no source of income at
all. (Note that public assistance/welfare, social security, and child support are among the income types in
both surveys.) Thus, while most families do not receive income from all sources, we would also not
expect a family to have received income from no source, and for this reason their income value was
treated as invalid.
The amount of missing/null data in the two surveys differs quite notably on the surface; NLSY97
has quite a bit more invalid total income data than the CPS does. (Nearly 22 percent of total income data
for the families of NLSY97 youths are missing or zero, while only 2 percent of total income data for
families of CPS youths are null.) However, this can be partially explained by the fact that the CPS
imputes missing data, while the NLSY97 does not. The CPS imputation is two-fold. First of all, if a CPS
respondent does not indicate whether he or she received income, that indicator variable is imputed.
Second, if a respondent is coded as receiving a particular type of income but gives no value for that
income type, this value is imputed. Thus, the 2 percent of CPS total income data that, for the purposes of
these analyses, we consider missing come solely from families that were coded as receiving no income at
all. Conversely, the 22 percent of income data for the NLSY97 that is considered missing in these
analyses is made up both of families indicating they received no income (55.5 percent of the missing
cases) and of those who did receive income but refused to provide the amount or a range into which the
amount fell (44.5 percent of the missing cases). In order to further equalize the comparisons between the
two surveys, we identified imputed total income values in the CPS, removed them from analysis, and ran
the comparisons based on reported data only20. (Incidentally, only an additional 3.4 percent of the data
were imputed, resulting in a total 5.4 percent missing/zero total income rate for the CPS.) The section
describing the overall analysis discusses the results of this additional analysis.
In a further effort to reduce the effects of the differences in the amounts of missing data and the
way they are treated, we have provided two percentage columns in each of the attached tables. The first
of the percent columns displays overall percentages of youths with families in each income range, with a
cell indicating the percentage of missing/null income data. The second column shows valid percentages,
giving the percent of youths with families in each income category based solely on those families with
valid income data. These valid percentages allow us a more accurate comparison of NLSY97 and CPS
data.
20
The CPS provides imputation flags for every source of income collected. In order to code cases as imputed or not
for a particular income type, a set of rules was necessary. See the next section, “Construction of the Estimates,” for
details.
143
While all four income categories are displayed in the tables that follow, we will focus our
analysis on the final category, total income from all sources. The reader may make other comparisons on
his or her own; however, we urge caution, especially with the self-employment income category where
the limited sample sizes cause sampling variability to be enormous.
Construction of the Estimates. In this section, we present a detailed description of how each
estimate was constructed. As described throughout this document, four variables were constructed for
comparison between the NLSY97 and the CPS; these variables include wages and salary income, selfemployment income, total earned income, and total income from all sources. Each of these variables was
created for the NLSY97 and the CPS.
For the NLSY97, up to three income questions created the basis for the final variables used in our
analysis. The first of these asked if the respondent received any income from a particular source. If not,
no further questions were asked, and the raw income value for that income source was coded as zero. If
the respondent did receive income, the next question asked for a dollar value for that income type. If the
respondent provided a dollar value, that value was saved as the raw income value for that variable. If not,
a final question requested the respondent to provide the range into which his or her income from that
source fell. If the respondent provided a range, then the midpoint of that range was saved as the raw
income value for that source. If no range was given, then the raw income value was set to missing for
that source. Next, the sources were combined to form the four variables used for comparison. Wages and
salary income only includes income from the wages and salary source. Self-employment income includes
income from both farm and non-farm self-employment. Total earned income includes wages and salary
income and self-employment income. Finally, total income from all sources is the sum of total earned
income and income from interest, AFDC, food stamps, supplemental security, child support, and all other
income. Once these income values were summed for each of the four income types, the raw values were
converted to ranges for comparison with the CPS. The eight range categories are: None, $1-$5,000;
$5,001-$10,000; $10,001-$25,000; $25,001-$50,000; $50,001-$100,000; $100,001-$250,000, and
Missing. For individual income sources, “None” is a valid value. However, as described earlier, if total
income from all sources has a value of zero, this is considered invalid. Only in the case where all
contributing income sources are missing is the actual value for a particular income variable coded as
missing. (For example, if the wages and salary source had a missing value, then the wages and salary
income variable would be coded as missing. In order for the total income from all sources value to be
missing, though, every one of the sources included would have to be missing.) Tables G-1 through G-8
and G-17 through G-20 display the income variables for the NLSY97 overall, for the cross-sectional
sample, and by race/ethnicity.
The CPS variables were constructed in an almost identical manner, with only a few minor
differences. First of all, while the NLSY97 had three potential questions for each income source, the CPS
had two. The respondent was asked whether he or she received income from a particular source, and
then, if so, what the dollar value of that income was. There was no range question asked in the CPS.
However, as described earlier, in the case of missing data, the values were imputed. Thus, if the
respondent did not report whether he or she received income from a particular income source, that
variable was imputed. If the imputed value indicated that income was received, then the income (dollar)
value was also imputed. Likewise, if the respondent did report having received income from a particular
source but the actual value was not reported, then that value was imputed. (For most of the analyses that
follow, CPS data using imputed values for missing data was used.) Finally, the four comparison variables
were constructed just as the NLSY97 variables were constructed, converting all raw values into the eight
range categories.
144
In order to conduct an analysis based on reported CPS data only (excluding all imputed
information), we constructed four additional variables for the CPS. Because there were many ways in
which some of the income types could be classified as imputed, we constructed a set of rules for this
classification procedure. The following were the rules we used in identifying imputes for each income
variable. For the first three income types: if the portion which made up the majority of that income type
was imputed, then the case was coded as imputed. (For example, if total earned income was made up of
60 percent wages and salary, and 40 percent self-employment income, and if the wages and salary income
value was imputed, then the total earned income variable would be classified as imputed. If the wages
and salary income was not imputed, but the self-employment income was, this case would not be
classified as imputed.) For total income from all sources, the process was two-fold. If the majority of
total income was earned, then if the portion which made up the majority of the total income was imputed,
the case was coded as imputed. Otherwise, if the majority of income was not earned, then if one or more
types of unearned income was imputed, the case was coded as imputed. (For example, if the majority of
income was earned and came from wages and salary, and wages and salary was imputed, then the variable
would be coded as imputed. Further, if the majority of income was not earned, but income that was
received from an unearned source such as public assistance was imputed, then the variable would be
coded as imputed.)
This calculation of the variables excluding imputed values for the CPS resulted in only 3.4
percent of the data being coded as imputed. This, in addition to the 2 percent of CPS families which
reported no income from any sources results in a total 5.4 percent missing or invalid total income rate. As
a note of warning, though, please be aware that other authors have cited much higher imputation
percentages for the CPS; these authors have considered an income value for which any portion was
imputed to be imputed, while we have used a majority rule.
Overall Analysis. The comparison data for CPS and NLSY97 are summarized in tables G-1
through G-24. The results of these comparisons indicate that, overall, there are some differences between
the NLSY97 family income distributions and the CPS family income distributions; however, these
differences are most obvious in the race/ethnicity (particularly Hispanic) breakouts.
Tables G-1 through G-4 and G-9 through G-12 display the NLSY97 (full sample) and CPS
youths’ family income distributions for each of the four income categories. In comparing these
distributions, there are small but significant differences between the two groups. For total income (tables
G-4 and G-12), both NLSY97 and CPS families tend to be concentrated near the center of the income
distribution, the majority falling into the $25,001 to $50,000 and $50,001 to $100,000 ranges. However,
NLSY97 has more families than CPS in the lowest income categories, slightly more than CPS in the
middle ranges, and fewer than CPS at the higher income levels. The following graph illustrates the
similarities and differences in total family income between NLSY97 and CPS.
145
Figure G-1 Percentages of Youths within
Income Categories, CPS vs. NLSY
35.0%
30.0%
25.0%
20.0%
CPS
15.0%
NLSY
10.0%
5.0%
0.0%
Total Income
While significance testing is generally inappropriate for sample sizes of this magnitude due to the
tendency for even small differences to become greatly exaggerated, and although the criteria for a formal
test, including simple random sampling, are not met here, a t-test was performed to compare the means of
the NLSY97 and CPS distributions and their variances. At this writing we have not yet estimated the
design effects for the NLSY97 sample, but we are certain that they will be greater than one. The t-test
calculated here incorporates a design effect borrowed from the full NLSY79 sample of youths (deff
approximately equal to 2). The results of the test show NLSY97 with a significantly lower mean total
income than the CPS. The families of NLSY97 youths have a mean total income of $47,199, while
families of CPS youths have a mean total income of $53,967. The t-value associated with the difference
in means, taking into account the design effect, is 5.1, which is significant at the 0.01 level; still, the
variances of these distributions were found to be equal. We must emphasize that these samples are quite
large, and with any sample large enough many differences appear to be significant. However, regardless
of the inadequacy of this test under these circumstances, it does give at least some evidence that these
distributions are not identical.
A second analysis was conducted to compare the raw CPS income distribution to the NLSY97
distribution by removing all imputed CPS income values. Tables G-13 through G-16 display these
revised weighted CPS income distributions. A second t-test was also run in order to compare the revised
mean total income to the NLSY97 mean total income. As mentioned earlier, only 3.4 percent of CPS
cases had total income data imputed, and an additional 2 percent claimed to have received no income at
all; thus, a total of 5.4 percent of the CPS cases were left out of this analysis. The results of the t-test
were almost identical to the results seen above. The NLSY97 again displays a significantly lower mean
income ($47,199 compared to $54,637 for the CPS), and the t-value associated with this difference was
5.6, also significant at the 0.01 level. Because these results are so close to the original analysis for total
income, we have not conducted any other analyses based solely on reported CPS data.
Cross-sectional Analysis. The NLSY97 cross-sectional sample behaves similarly to the overall
sample. Tables G-5 through G-8 display cross-sectional income distributions for the four income
146
categories, and valid percentages are almost identical to those of the overall sample. As an example, table
G-8, which displays total income percentages for the cross-sectional sample, parallels the overall sample
in its comparison to the CPS total income distribution: families of NLSY97 cross-sectional youths are
more prevalent in the lower income categories (11.7% of families of NLSY97 youths earn between $1
and $10,000 compared to 9.2% of families of CPS youths), similarly distributed in the middle income
groups (62.0% of NLSY97 youths’ families and 62.5% of CPS youths’ families fall into the $25,001$100,000 range), and less prevalent in the upper income categories than CPS (7.2% of NLSY97 youths’
families earn more than $100,000, compared to 9.4% of CPS youths’ families).
Race/Ethnicity Analysis. Race breakouts, displayed in tables G-17 through G-24, show a fairly
consistent picture, across variables, between NLSY97 and CPS. Hispanics have the most notable
differences; NLSY97 has more families than CPS in the lowest income extreme (9.6% of NLSY97
Hispanic youths’ families vs. 4.6% of CPS Hispanic youths’ families earn $5,000 or less), more in the
upper-middle and upper ranges (for NLSY97, 22.4% earn more than $50,000, compared to 16.6% for
CPS), and fewer in the lower and middle (68.0% of NLSY97 Hispanic youths’ families earn between
$5,001 and $50,000, versus 78.7% of CPS Hispanic youths’ families). Family incomes of NLSY97 black
youths are distributed approximately the same as for the CPS, with slightly more at the lower end (10.7%
in the $5,000 or less range for NLSY97, compared to 8.3% for CPS) and slightly fewer at the higher end
(2.2% in the over $100,000 range for NLSY97, compared to 3.1% for CPS). Finally, NLSY97 families of
non-Hispanic, non-black youths were also very similar to their CPS counterparts. While more do fall into
the lower income groups (8.3% earning less than $10,000 for NLSY97, compared to 5.5% for CPS) and
slightly fewer into the higher income groups (8.9% earning more than $100,000 for NLSY97 versus
11.9% for CPS), the majority are distributed similarly among the middle income groups.
Summary. Given sampling variability and definitional and procedural differences, these
comparisons are encouraging. The CPS and NLSY97 total income distributions have equal variances,
and although the test conducted on the overall sample does show a significant difference between the
mean incomes of the two populations, it does not necessarily indicate an important discordance; sample
sizes of this magnitude very frequently result in significant results for even minor differences.
Furthermore, the income distributions do not look considerably different to the naked eye. The most
visible distinction may be that the NSLY97 distribution looks as if it were the CPS distribution shifted
slightly to the left. The results of these analyses were also consistent across sample subdivisions; findings
were similar by race/ethnicity and for the cross-sectional compared to the overall sample. Thus, in
closing, while this study does find significant differences between the means of the total income
distributions, individual users must judge for themselves whether this difference is substantively
important.
As one additional note of warning, we would like to draw attention to the “None” category for
each of the income types, specifically for total earned income. According to tables G-3 and G-11, nearly
21 percent of the NLSY97 sample responded that they did not earn any income that year, compared to
only 8 percent of CPS cases. On common sense grounds, it seems quite unlikely that such a large
percentage of the population would be living on interest and dividends or public welfare. Thus, we
suggest that the “None” category may in some cases be another form of refusal; some respondents may
say that they received no income in order to move through the interview more quickly. In comparing
tables G-3 and G-11, one can see that the major difference between the CPS and NLSY97 samples lies in
this category; the remaining income category percentages are much closer to each other. Thus, users of
our dataset must be aware of these “None” cases and have a strategy in place for dealing with them.
147
APPENDIX H
Introduction and Paper Screener
156
NLSY97 SCREENER INTRODUCTION
Good morning/afternoon/evening!
My name is __________, and I am in your neighborhood today working on a national study for the
United States Department of Labor, Bureau of Labor Statistics. I am a field interviewer for the National
Opinion Research Center at the University of Chicago (show identification). I understand that you
received a letter about this important study a few days ago? (HAVE A COPY OF LETTER AND
BROCHURE READY TO PRESENT IF RESPONDENT INDICATES LETTER NOT
RECEIVED; ALLOW A MOMENT IF HE/SHE WANTS TO GLANCE AT THEM.)
Now, before I start the interview, I’d like to tell you some important information about this study.
This study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics, under authority of
Title 29 USC 2. Your participation is vital to the success of the study and is voluntary. This survey is
conducted to understand the labor market experience of young adults. This information will be used by
BLS and other researchers. The information you provide is protected under the Privacy Act. Personal
identifiers, such as name, address, or Social Security numbers will be held in confidence and will not be
released to the public.
1.
First I’d like to do a 3 minute interview to determine if any persons in this household are eligible
for the survey. Let’s begin by asking a few questions about the people who usually reside in this
household, including persons:
-
who are away at school or college
lodgers, boarders, or persons in your employ who live here
any others who usually live here but are away at present, such as someone travelling, or someone
in a hospital, correctional facility or any other type of institution
any others who are staying here such as foster children
and any babies or small children
-
START PAPER SCREENER. GET NAME FOR EACH HOUSEHOLD MEMBER.
2a. Could you tell me the names of the people who usually live here, beginning with the eldest?
(ENTER NAMES, INCLDING NICKNAMES/INITIALS IN COLUMN 1. CIRCLE NAME OF THE
HOUSEHOLD INFORMANT.)
2b. I have the following people listed as living here (READ NAMES IN COLUMN 1.)
Have I missed anyone? (ADD NAMES.)
NEXT, GET DATE OF BIRTH FOR EACH HOUSEHOLD MEMBER: CONTINUE
WITH Q.3 ON REVERSE SIDE.
- OVER -
157
3. Now I would like to ask you about the ages of the people you have just mentioned.
3a. What is ____________ date of birth? (READ FIRST/NEXT NAME)
(ENTER D.O.B. IN COLUMN 2–ENTER EVEN PARTIAL INFORMATION SUCH AS THE
YEAR OF BIRTH–AND PUT AN “X” IN COLUMN 4, 5, OR 6 AS APPROPRIATE.)
ASK 3b. IF DATE OF BIRTH INFORMATION IS INCOMPLETE FOR THAT NAME.
OTHERWISE, REPEAT 3a. FOR NEXT NAME, THEN GO TO 4 WHEN D.O.B.
INFORMATION IS COMPLETE FOR ALL NAMES.
3b. What is __________ age? (READ FIRST/NEXT NAME) (ENTER AGE ON THE
APPROPRIATE LINE IN COLUMN 3 AND PUT AN “X” IN COLUMN 4, 5, OR 6 AS
APPROPRIATE.)
-- IF THE ANSWER TO 3b. IS ANY AGE OTHER THAN 17….…………….GO TO 3a.
-- IF THE ANSWER TO 3b. IS 17………………………………………………GO TO 3c.
-- IF THE INFORMANT DOES NOT KNOW AGE OR HESITATES……...…GO TO 3d.
3c. Has _______________ (READ FIRST/NEXT NAME) had a birthday since January 1st 1997?
(MARK “Y,” “N,” OR “DK” NEXT TO THE NUMBER 17 IN COLUMN 3, AND GO TO 3a.)
3d. Would you say that ___________ (READ FIRST/NEXT NAME) is 12 to 17 years old?
(IF YES, WRITE “12-17” IN COLUMN 3, AND MARK AN “X” IN COLUMN 4; IF NO ASK)
Would you say that ___________ (READ FIRST/NEXT NAME) is 18 to 23 years old?
(IF YES, WRITE “18-23” IN COLUMN 3, AND MARK AN “X” IN COLUMN 5; IF NO ASK)
Would you say that ___________ (READ FIRST/NEXT NAME) is
- under 12 years old?……….(IF YES, WRITE “UNDER 12” IN COLUMN 3)
- 24 to 31 years old? …….....(IF YES, WRITE “24-31” IN COLUMN 3)
- 32 to 40 years old? ……….(IF YES, WRITE “32-40” IN COLUMN 3)
- 41 or older?………………..(IF YES, WRITE “41 OR OLDER” IN COLUMN 3)
GO BACK TO 3a. UNTIL D.O.B. INFORMATION IS COMPLETE FOR ALL NAMES, THEN GO TO 4
LIVING IN THE U.S. QUESTION
4. FOR EACH PERSON WITH AN “X” IN COLUMN 6, ASK:
(Were/Was) you/name living in the United States in 1978?
(CIRCLE THE Y, N, OR DK AS APPROPRIATE IN COLUMN 7.)
CHECK FOR ELIGIBLE HOUSEHOLDS
5a. INTERVIEWER: IF THERE IS AN “X” MARKED IN COLUMN 4 OR COLUMN 5, SAY:
It appears that members of your household are eligible to participate in our study. I will need to enter some
information into my laptop computer, and continue the interview. May I come in to do that or if you prefer, I will
proceed from out here. TRY TO COMPLETE FULL SCREENER ON CAPI.
5b. INTERVIEWER: IF THERE IS NO “X” MARKED IN COLUMN 4 OR COLUMN 5, SAY:
It does not appear that anyone in your household will be eligible to participate in this study, but I would like to
thank you for taking the time to speak with me. My supervisor randomly checks my work, and someone MAY call
to verify that I was actually here. Could I please get your telephone number for that verification call? (ENTER
PHONE NUMBER ON PAPER SCREENER FORM.) Thank you again. Goodbye.
158
Date ___________________________________
Time ___________________________________
NLSY PAPER SCREENER
HOUSEHOLD ID #
PSU: _______________
FI NAME _____________________
SEG: _______________
FI ID#: _______________________
LINE#/ADDRESS: ________________________________________
________________________________________
CITY/STATE/ZIP: ________________________________________
(1)
Name: First/Last
Interviewer: Circle
Household information’s
Name)
(2)
DATE OF
BIRTH
(D.O.B)
(3)
AGE
(4)
Interviewer: Put
an X in box if
D.O.B between
1/1/80 and
12/31/84 or age
12 through 17
(5)
Interviewer: Put
an X in box if
D.O.B between
6/1/73 and
12/31/79 or age
18 through 23
(6)
Interviewer: Put
an X in box if
D.O.B between
1/1/57 and
12/31/64 or age
32 through 40
(7)
Interviewer: If X in
column 6 ask:
“(Was/were)
you/name living in the
United States in
1978?
(Circle Response)
APPOINTMENT DATE: ________________________
TIME: _____________
Y
N
DK
Y
N
DK
Y
N
DK
Y
N
DK
Y
N
DK
Y
Y
N
N
DK
DK
Y
N
DK
Y
N
DK
Y
N
DK
TELEPHONE NUMBER: ________________
OTHER INFORMATION: __________________________________________________________________________________
159
APPENDIX I
Introduction and Age Questions from
Extended Screener
160
APPENDIX L
Review of Gatekeeper Interviews
177
Introduction
This report summarizes the results of reviewing a statistical sample of the records of calls (ROCs)
for cases given a field disposition of 71, “gatekeeper,” by NORC Field interviewers. There were 4,055
households identified as “gatekeeper” during the data collection and 494 were reviewed. The purpose of
the review was to ensure that the proper procedures were followed and that there were no indications of
youths in the households due to improper use of this disposition code.
During the pretest of Round 1, NORC and BLS realized that many households were extremely
hard to reach, especially those in multi-unit buildings, locked buildings or gated communities. In
addition, it was found that many of these types of units were housing for seniors or that in order to live
there household occupants had to be at least 55 years of age and children were not allowed to live there as
residents. Field interviewers made multiple visits and still were unable to get past security guards, call
boxes and building managers; hence, we used the term gatekeeper. NORC and BLS needed to find more
cost-efficient ways to determine if there were eligible youths in these households. After the Round 1
pretest, NORC and BLS decided to allow the field interviewers to receive help from gatekeepers (persons
who blocked the field interviewer from talking to a respondent) in determining whether any youths lived
at the sample households. While there was worry about misuse of this procedure, this review shows that
there were few cases where errors were found.
Rules for Using the Gatekeeper Code
To use the gatekeeper code, the field interviewer must have had no contact with anyone in the
selected housing unit. They were to try everything possible to get into the building. However, if a
building manager or other building gatekeeper absolutely refused entrance, and the building manager told
them it was a “seniors” building or some other such statement, the field interviewer had to get permission
from their field manager before assigning the gatekeeper code.
However, if the field interviewer had any indication that there were youths within the specified
age range, or the gatekeeper did not know for sure if there were youths in the household, the field
interviewer was to continue his or her efforts to contact the sampled household. If they were still
unsuccessful after having done all they could do to contact the household, they were to consult with their
field manager to determine if they should end their attempts to contact this household. If, by talking to
the field interviewer, the field manager determined that nothing else could be done and the case should be
finalized, the field manager would assign a final noninterview code.
178
Records of Calls
NORC uses a Case Management System (CMS) to control the disposition of cases. In the CMS,
the field interviewers have the ability to enter information about efforts to get an interview. They are not
required to enter a record of call (ROC) for every attempt, phone call or visit. For example, they may go
by a house several times in one day, but only enter one ROC. Therefore ROCs can’t be used to determine
number of contacts. The field interviewers were not told to enter the number of household members or
whether there were youths. However, many field interviewers did in fact enter information that some
number of youths were in the household, even when they were unable to get a completed interview. They
tended to make statements like, “I saw teenagers coming out of the house,” “a neighbor said there were
eligible youths in the household,” or “a neighbor says only an 80-year old woman lives there.” The fact
that they did not enter such information does not mean they made a mistake since there was no
requirement to do so, just as there was no requirement to enter a ROC for every contact with the
household. It was thought to be too time consuming to enter a ROC for every visit, since they often stop
by a sample address several times a day when they are in the area. Opening and closing the case in a
laptop computer takes valuable time. In future screenings we would recommend that they be required to
enter any possible information about household members in a note in the ROC and to enter a ROC for
every household contact. This information is very useful when reviewing information about the sample.
Sample
The 4,055 gatekeeper screeners were sorted by field interviewers with increasing numbers of
gatekeeper cases. Five samples of 98 or 99 cases each were drawn. Starting with case one, using a
selection interval of 41, the first sample was selected. This process was repeated until five independent
samples were drawn. The purpose of drawing five samples was to decrease the work of the person
reviewing the ROCs. If no problems were found in the first sample, then review of the other samples
might not be necessary. However, the reviewer decided to review all five samples to be certain there
were no problems. As a result, 494 screener ROCs were reviewed.
Results of Review
A review of the statistical sample of 494 households given a code 71, gatekeeper, found that one
household had possible youths in the 17- to 23-year-old age range (ASVAB eligible). Two households
were found that had one or more eligible youth in the 12- to 16-year-old age range (NLSY97-eligible). If
you extrapolate to the total number of 4,055 gatekeepers, one would estimate eight ASVAB-eligible
households and 16 NLSY97 households. Slightly over 69 percent of the sample had specific notes in
either the ROCs or the Noninterview Report that there were no eligible youths in the household and
approximately 29 percent had no such statement, but there were also no notes indicating youths were
present. Since such a note was not required, one should assume these were done correctly and there are
no youths. Previous reviews of ROCs have indicated that field interviewers are much more likely to have
written a note when there were youths than when there were not.
Slightly over 19 percent of the time, the field interviewer gave a disposition of gatekeeper when,
in fact, a household respondent told her/him that there were no eligible youths. These cases should have
been coded as refusals. If one extrapolates to the full sample, there would have been approximately 788
such cases.
Slightly over one percent of the sampled cases should have been coded as vacant, demolished or
some other type B or C noninterview.
179
The majority of the time (78.5 percent), field interviewers correctly followed the procedure. In
those cases where procedures were not followed, only 0.6 percent were households with possible youths
present.
Table L-1 Review of Gatekeeper Record of Calls
Number of
Households in
Sample
Percent of
Households in
Sample
Total
494
100
No kids, definitive statement***
No definitive statement, but no kids indicated***
Household respondent said no one eligible****
Housekeeper, babysitter said no one eligible
No kids, crash
No kids, should be vacant
No kids, should be Type C (demolished, etc.)
No kids, bad address
Possible age 17-23 in household
Possible age 12-16 in household
240
148
96
0
1
4
1
1
1
2
48.58
29.96
19.43
0.00
0.20
0.81
0.20
0.20
0.20
0.40
Description*
*Interviewers were not instructed to write any particular type of note in the record of calls, so the
fact that they did not indicate youths does not mean they erred.
***About 2-3 percent of these should have been a proxy screener and not a gatekeeper. Even
though it was generally in an apartment building, they went to a neighbor, so therefore had gotten
in the building. The gatekeeper code was reserved for inability to get into a building or
community due to locked doors.
****Technically these should have been refusals. Once a field interviewer talked to a household
respondent, it was no longer a gatekeeper case.
180
APPENDIX M
NIR Reports, NLSY97 Round 1
181
1. Introduction21
This report summarizes analyses of Non-Interview Respondent (NIR) Report data for households
selected for NLSY97 screening. NIR reports were used during the Round 1 screening effort to document
screener cases in which interviews were not completed during the screener field period from January
through October of 1997. NORC Field Interviewers finalized a case as an NIR only with Field Manager
permission. Out of 90,957 sampled housing units, NORC completed screeners with 75,410, including
screeners completed with household respondents or with gatekeepers and proxy respondents. NIR reports
were completed for 13,810 of the 15,547 non-interviewed housing units. Among these 13,810 NIRs are
10,436 units which were either vacant or uninhabitable at the time of the screening interview. Thus, the
NIR reports offer information on 3,374 sampled housing units that were eligible for the screener but were
finalized as non-interviews. The purpose of the NIR Report analysis is to obtain some impressions of the
non-interview housing units in our screening sample, to highlight possible trends among screener
nonrespondents, and to identify potential sources of bias in our youth sample. Figure M-1 shows the
number of households that completed screeners, NIR reports, and the reasons they provided for noninterviews.
21
Final updating of the NLSY97, Round 1 disposition codes was done in November 1999, in light of all evidence
on eligibility collected to that point in time, including certain evidence collected during Round 2 operations.
Because the tabulations cited in this appendix reflect an earlier generation of the Round 1 disposition codes, there
may be small discrepancies between counts cited here and the final counts cited in the main body of this NLSY97
Technical Sampling Report.
182
Figure M-1 Number of Cases: Screeners and Non-interviews
Sampled Housing Units
90,957
Completed Screeners
75,410
Did not complete screener
15,547
Includes proxy and
gatekeeper cases
Completed NIR reports
13,810 HUs
Vacant/
Uninhabitable
HUs
10,436
Vacant, under
construction
7,417
Not usable as a
residence
443
Business
231
Not a permanent
residence 2,435
183
No NIR report
1,737
Non-interviews,
eligible for
screener
3,374 HUs
Refusals
1,556
Technical Problems 610
Inaccessible
144
Language Barrier
178
Ill/ Handicapped
15
Field period ended/ R
never home
871
2. Purpose of Non-Interview Report
For every sample housing unit, Field Interviewers (FIs) attempted to complete a household
screener -- a roster of all usual residents of the sampled address, including their age, sex, race and
ethnicity. When potentially eligible NLSY97 youths or PAY97 youths were identified on the basis of age
and ethnicity data, the Field Interviewer then attempted to complete an Extended Screener, which
confirmed youth eligibility by collecting school enrollment information and gathered additional
background information about the household members. Finally, the Field Interviewer attempted to
complete a parent and youth interview or secure test participation.
When a household refused to participate in a screener interview or a screener was not completed
for some other reason, the project failed to identify and interview eligible youths. Even impressionistic
data about omitted households can be valuable in assessing the screener effort. The NIR report data are
important for identifying potential sources of bias in our sample that we can then investigate more
rigorously.
3. NIR Data Collection
3.1 The Instrument
The NIR was administered using computer-assisted data entry technology (CADE) and was
completed by the Field Interviewer at home after a case had been finalized as a non-interview. The
CADE instrument comprises three main sections, addressing the following questions: (1) Why were we
unable to complete a screener? (2)What type of households were non-interviews? (3) How many
potentially eligible youths were missing from our sample? Table M-1 lists questionnaire items for each of
these main questions.
184
Table M-1 NIR Sample Questions Divided Into Three Main Sections
(1) Why were we unable to complete a screener?
Χ What is the reason for NIR?
Χ
If refusal, what methods were used to convert the respondent?
Χ Did the respondent / informant ask any of the following questions?
Χ Did the respondent / informant make any of the following comments?
(2) What type of households were non-interviews?
Χ
What kind of housing unit is this?
Χ
How would you describe the immediate area or street (one block, both sides) where this
housing unit is located?
Χ
What type of residence is most common on the street (one block, both sides) where this
housing unit is located?
Χ
How well kept are most of the buildings on the street where this housing unit is located?
Χ
How well kept is this housing unit?
Χ
What is the sex of the household member / informant?
Χ
What is the race / ethnicity of the informant / respondent?
(3) How many potentially eligible youth were missing from our sample?
Χ How many people do you think live in this housing unit?
Χ
How did you determine the number of people living in this housing unit?
Χ
Do you think anyone between the ages of 10-25 years of age lives here?
Χ
How many people between the ages of 10-25 years live here?
Χ
Do you think any are 17-25 years of age?
Χ
How many people between the ages of 17-25 years live here?
Χ How did you determine this?
3.2 Respondent vs. Informant
Respondents were the people who were eligible to complete a youth or parent questionnaire or the
household screener (and as appropriate, the extended screener). A screener respondent was someone 18
years of age or older who usually resided in the household and who was a responsible adult. However, if
there was no such respondent available at the household, an interviewer could have spoken to an
informant outside of the household who was a responsible person 18 years of age or older and who was
familiar with the household. Typical informants outside the household would be visitors to the household,
neighbors who live next door or directly across the street, the postal carrier or the doorman for the
building. Informants were sought by field staff to assist in securing the cooperation of respondents,
complete proxy or gatekeeper reports, or provide information for the NIR report.
The questions in the NIR regarding reasons for non-interview, housing units, neighborhood, and
number of household members or youths reveal information about respondent(s) only. However, the
remaining items regarding demographics and comments or concerns about the survey refer to either an
informant or a respondent, depending on who spoke with the interviewer. Details on respondents and
other informants can be found in the body of this report.
185
4. Rules for Completing a Non-Interview Report
Field interviewers were required to get the permission of a Field Manager before finalizing a case
as non-interview. The rules for completing an NIR differ according to whether the interviewer spoke
with a household respondent or non-household informant.
4.1 Household Informant / Respondent
To finalize a case as non-interview for a Household Respondent, the Field Interviewer must have
contacted a household member and must have attempted to complete a screener. For example, the
Household Informant may have (1) refused to complete the Household Screener (including breaking off
partway through the household screener interview) (2) completed the household screener but refused to
participate in the extended screener when potentially eligible youth were identified. If the respondent
refused, then the Field Interviewer attempted a conversion. After repeated unsuccessful conversion
attempts, a Field Manager would designate the case as a final screener non-interview. Youths were only
spawned when the entire screener was completed.
4.2 Non-Household Informant
If no contact was made with a household respondent, then the Field Interviewer sought to obtain a
proxy screener or a gatekeeper screener. These were data collection alternatives when the residents of the
housing unit were exceedingly difficult to locate and were used only to identify a housing unit as having
no potentially eligible youths or as a source of NIR Report but not of screener data.
A case was assigned a proxy screener code when an acceptable proxy respondent was able to
definitively state that no household members were between ages 8 and 28 years. When a proxy screener
could not provide this confirmation, then the interviewer continued to seek a screener respondent from the
sampled unit. If still no screener respondent was available, the cases were coded as final non-interview if
permitted by the Field Manager.
A case was assigned a final gatekeeper code when the interviewer was blocked from talking to a
respondent and was able to obtain residential eligibility documents which confirmed that no youths aged
8 to 28 lived there. When eligibility documents did not provide this confirmation, the Field Interviewer
continued to attempt screening. If no screener respondent was available, the case was coded as a final
non-interview if permitted by the Field Manager.
Even if a Field Interviewer was unable to obtain an acceptable proxy screener or gatekeeper, a
Non-Household Informant might have been sought to assist in the completion of an NIR report or to gain
cooperation of the household. Such informants could not be used to complete a Screener and they were
only obtained for providing information on the NIR when permitted by a Field Manager. Although the
informant’s information was used in the NIR report, the screener case itself was designated a ‘refusal’ or
as some other type of non-interview and we did not use the informant’s information in identifying the
sample or in our analyses of screener data.
186
5. Results of the NIRs
5.1 Why Were We Unable to Get an Interview?
After gaining the permission of a Field Manager to finalize as non-interview, Field Interviewers
gave a final disposition code to the case to explain the main reason they were unable to obtain an
interview. For over 75 percent of the 13,810 NIR reports, Field Interviewers were not able to complete a
screener because the sampling unit was not a household unit or because it was a vacant unit--3 percent not
usable as a residence, 2 percent business address, 17 percent not permanent residence, and 54 percent
vacant or under construction. The next most frequent reason for no interview is a refusal (including
break-offs or partial screeners)--11 percent of non-interview housing units resulted because a Household
Respondent refused to complete a screener. Table M-2 shows the distribution of reasons for an NIR.
Table M-2 Reasons for a Non-Interview
Description
Number of
Households
with NIR data
610
Percent of
Households
with NIR data
4.4
Field Period Ended / Respondent never home
871
6.3
Not Usable As Residence
443
3.2
Business--not a household unit
231
1.7
Not a Permanent Residence (i.e. Vacation home)
2,345
17.0
Vacant / Under Construction
7,417
53.7
Household Unit Inaccessible
144
1.0
Language Barrier Cases
178
1.3
15
0.1
1,556
11.3
13,810
100.0
Technical Problems
Respondent too ill / handicapped
Refusal
Total
5.2 Refusals
First we consider the subset of NIR reports where refusals were the reason for non-interview.
There were 1,556 such households. When a respondent refused, Field Interviewers attempted to “convert”
the respondent and secure permission to complete a household screener and an interview when
appropriate. Conversion methods included sending a refusal letter or requesting a trained converter to
contact the respondent/informant by phone or in person. Table M-3 shows a comparison of conversion
methods and some general concerns expressed by the respondent/informant for households with
potentially eligible youths between the ages of 10 and 25 and for households without potentially eligible
youths.
187
Table M-3 Comparison NIR Reports Due to Refusals (1556
Households Out of Which 789 are Missing Data on the Number of Potentially Eligible Youths)
Percentage of Households
With No Potentially
Eligible Youth (91)
(74 Households, 17 missing)
Refusal Letter Sent
Attempt By Phone Converter
Attempt By Personal Converter
Attempt By Supervisor
Not Applicable
Other
Respondent/Informant Comments
54.1
8.1
64.9
2.7
14.9
10.8
(78 Households, 13 missing)
Percentage of Households
With Potentially Eligible
Youth (676)
(607 Households, 69
missing)
61.9
20.6
69.7
5.4
9.2
6.9
(598 Households, 78 missing)
I’m too busy / survey too long
I don’t like / do surveys
Don’t know anything about this
Not the sort of person you want
I’m not interested
This is too personal
Survey is a waste of money
Government has no business
Government already knows this
Don’t want children bothered
Let me think about it
Negative comment or delay
None of the above
Did the informant ask...
What is the purpose of the survey?
Who is the sponsor of the survey?
How was I chosen?
How long will the interview take?
Who sees answers?
Can I get a copy of the results?
Is there an incentive?
Informational question or
expression of interest?
17.9
28.2
3.8
1.3
41.0
7.7
3.8
5.1
1.3
7.7
0.0
12.8
29.5
(78 Households, 13 missing)
9.0
2.6
3.8
2.6
0.0
0.0
0.0
29.4
20.9
7.4
4.5
52.5
11.0
4.7
10.4
6.5
27.6
6.9
16.9
12.0
(592 Households, 84 missing)
35.8
22.3
28.7
15.4
9.3
0.5
1.4
1.3
2.5
Description
Method of Conversion
Overall, Table M-3 shows that Field Interviewers made additional efforts to complete a screener
where youths were thought to be present. Of households with potentially eligible youths, 62 percent
received a refusal letter and 21 percent received a call by a phone converter compared to 54 percent
refusal letters and 8 percent phone converters for households without potentially eligible youths. Of
households with potentially eligible youths, 28 percent of respondents commented that they “don’t want
children bothered” compared to 8 percent of respondents in households with no such youths. (Note that
households without potentially eligible youths could include younger children as household members.)
188
An effective tactic for gaining cooperation is to engage the respondent in conversation.
Comparing questions asked by the informant/respondent suggests that interviewers had more extensive
communication with informants when there were potentially eligible youths present than in other
households. For example, many more households with youths wanted to know the purpose of the survey
(36 percent), the sponsor of the survey (22 percent), how he/she was chosen (29 percent), the length of the
interview (15 percent) and who sees the answers (9 percent) compared to households without youths-- 9
percent wanted to know the purpose, 3 percent asked about the sponsor, 4 percent asked how he/she was
chosen, 3 percent asked about the length, and 0 percent wanted to know who sees the answers.
5.3 What Type of Households Were Non-Interviews?
The racial/ethnic composition of the non-interview housing units is similar to the national
composition. Of the informants/respondents, 67.3 percent are Non-Hispanic White, 0.2 percent American
Indian, 7.9 percent Asian/Pacific Islander, 8.3 percent Hispanic, 14.2 percent Black/African-American,
and 2.0 percent other. The most prominent difference is that the Asian/Pacific Islander population is
over-represented in the total NIR’s (7.9 percent), mirroring a tendency toward higher non-response
among Asian Americans in general population surveys. The majority of informants were female.
When we compared households on the basis of whether they contained potentially eligible youths
(Table M-4), we found that all inhabited households for which we had NIR reports were similar to
households with potentially eligible youths for gender of the informant, race/ethnicity of informant, type
of respondents’ housing units, and type of respondents’ neighborhood. One notable difference was that
there were fewer Asian/Pacific Island or informants for households with potentially eligible youths (2.8
percent) compared to all households (7.9 percent). While a high non-response rate among Asians/Pacific
Islanders was not surprising, it appears that we were able to decrease this nonresponse rate for
Asian/Pacific Islanders with potentially eligible youths, perhaps due to greater efforts by the field
interviewers.
Table M-4 shows a comparison of the non-interview households where Field Interviewers were
able to identify potentially eligible youths aged 10-25 with all the inhabited NIR households (including
the households where they could identify potentially eligible youths). There were 3,374 inhabited
households, out of which they could identify potentially eligible youths in 928 households. The remaining
reports either did not provide a response to the question about the number of family members (686),
reported that no one lives in the housing unit (67), provided invalid or don’t know responses (1192),
reported no 10- to 25-year-olds (470) or didn’t know that there were any potentially eligible youths (31)
in the household.
As one might expect, households with potentially eligible youths were less likely to live in an
atypical or “other” type of housing unit (1.6 percent) compared to all households (5.3 percent). Similarly,
households with youths were less likely to be in an atypical or “other” type of neighborhood (0.4 percent)
compared to all households(4.3 percent).
189
Table M-4 All Inhabited Households Compared to Those Where
Potentially Eligible Youth Were Identified
Description
Gender of Informant
Male
Female
Race/Ethnicity
Non-Hispanic White
American Indian
Asian/Pacific Island
Hispanic
Black-African American
Other
Type of Housing Unit
Detached Single Family
Mobile Home Trailer
Single Family Row
Small Apt. or Condo
Large Apt. or Condo
Apt. Attached to Commercial
Structure
Other
Type of Neighborhood
Rural and Agricultural
Rural and Residential
Suburban, Residential
Urban, Residential
Urban, Mixed Retail
Urban, Mixed
Wholesale
Vacant Buildings
Other
All inhabited households
(3374)
(percent of non-missing
households)
(1664 Households, 1710
missing)
37.9
62.1
(1488 Households, 1886
missing)
67.3
0.2
7.9
8.3
14.2
2.0
(1861 Households, 1513
missing)
72.5
4.1
4.7
7.6
5.6
Households where potentially eligible
youths 10-25 were identified (928)
(percent of non-missing households)
0.2
5.3
(1885 Households, 1489
missing)
9.3
15.5
45.2
18.7
5.7
1.2
0.0
0.1
4.3
0.0
1.6
(796 Households, 132 missing)
(781 Households, 147 missing)
36.5
63.5
(706 Households, 222 missing)
71.7
0.1
2.8
7.4
16.3
1.7
(793 Households, 135 missing)
75.9
5.9
5.9
6.2
4.4
11.7
17.5
43.0
20.4
5.4
1.8
0.0
0.0
0.4
In general, most of the non-interviewed housing units were detached single-family homes (72.5
percent). The most typical housing units surrounding the sampling unit closely match those of the sample
housing units.
Field Interviewers described a large majority of the household units as either very well kept (47
percent) or well kept (32 percent), while very few of the units were characterized as poorly kept (8
percent). The condition of the housing units in the surrounding area closely resembled these sampling
units.
190
5.4 How Many Household Members/Potentially Eligible Youths Are Missing From Our Sample?
The 3,374 inhabited units among the NIRs contained a total of 4,289 household members. Of
greatest relevance to the NLSY97 and PAY97 samples are the number of potentially eligible youths
missing from our sample. Field Interviewers were able to estimate the number of household members for
1,429 households and the number of potentially eligible youths for 1,398 households. Other households
are not enumerated because the Field Interviewer was unable to secure an informant who could provide
household counts. Using the 1,398 enumerated households, we estimate that 928 had at least one
potentially eligible youth (10 to 25 years of age), and we have the number of youths for 923 households;
of these households with at least one potentially eligible youth, the average number of 10- to 25-year-olds
is 1.52. Households with at least one youth aged 17 to 25 years had on average 1.32 such youths. Data
from 923 households indicate that 1,401 potentially eligible youths were associated with these NIR
households and so omitted from the NLSY97 youth study. (The remaining 2,451 households were
missing data for number of youths or were reported to have no youths.) We can count 430 households
with 570 youths ages 17-25 from the enumerated households, and estimate that there were 831 youths
aged 10-16 years. The composition of the NIR households is shown in Table M-5. Figure M-2 shows
how the number of youths was determined.
Table M-5 Composition of NIR Households
Total number of NIR housing units which are inhabited
3,374
Total number of housing units where we have a count of household members
1,429
Total number of household members
4,289
Housing Units which are believed to contain a potentially eligible youth
1,398
Number of households with at least one 10- to 25-year-old
Number of youths aged 10-25
923
1,401
Average number of 10- to 25-year-olds per household with at least one 10- to 25year-old
Number of households with at least one 17- to 25-year-old
1.52
Number of youths aged 17-25
570
Average number of 17- to 25-year-olds per household with at least one 17- to 25year-old
Estimate of number of youths aged 10-16
1.32
430
1,401-570=831
In Table 5.24 of this report, we find that the CPS density estimates (for 12- to 23-year-olds) are
around 16 percent while the NLSY97 estimates are lower (around 12 percent). Our estimate of 10- to 25year-old youths as a proportion of total household members from the NIR reports is approximately 32
percent. Even though the age ranges do not exactly match, we do find the density of youths in the NIR to
be much higher, indicating that target youths constitute a larger percentage of the population in the noninterview group than among those completing screeners. This is consistent with the idea that households
with youths may have been more likely to refuse to be screened, causing the shortfall of sampled youths
in the NLSY97.
191
Table M-6 Ratio of Youths to Household Units and to Household Members
Description
NIR ratio
10-25 year olds to total number of household members
17-25 year olds to total number of household members
10-25 year olds to number of household units
17-25 year olds to number of household units
.32
.13
.42
.17
While making attempts to gain cooperation and complete the household screener, Field
Interviewers exploited a variety of tactics to determine the number of household members and the number
of potentially eligible youths. For 68 percent of the cases, Field Interviewers spoke with and even began
the household screener interview with the respondent before he/she refused and so they were able to
secure reliable information about youth residents in the household. Out of the 1,401 youths counted, 962
youths are counted from information given directly by the respondent. Of the remaining cases, the
number of youths in the sample households were determined by one of the following methods: asked a
neighbor (10.7 percent), asked the postal carrier (0.2 percent), saw someone come out of the house (9
percent), or other methods (11.6 percent). Table M-7 shows which method was used to determine the
number of potentially eligible youths for all 923 households which have at least one youth aged 10-25.
The remaining 2,451 inhabited households had missing household counts, invalid/don’t know responses
or said they had no youths aged 10-25.
Table M-7 Determination of the Number of Potentially Eligible Youths
Percentage of households
(923 households)
Method Used to Determine
Respondent Was Willing to Tell Me
Asked a Neighbor
Asked the Postal Carrier
Saw Someone Come Out of the House
Other
Don’t Know
68.0
10.7
0.2
9.0
11.6
0.5
Apart from the 1,401 youths aged 10-25 already identified, we may have missed youths either
because we have no NIRs for some cases or because the number of household members or youths was not
reported. Out of the non-interview households, the 1,737 missing NIR households, shown in Figure M-1,
could contain eligible youth. Also, informants often gave “don’t know” and “invalid” responses and for
some households there is missing information on the number of household members. These households
may have youths in the 10-25 age range too.
To calculate a range of estimates for youths ages 10-25, we assume that households for which the
reports had missing, “don’t know” or “invalid” information on the number of household members and
youths, could have contained some youths in the 10-25 age range. Figure M-2 shows the counts on which
this estimate is based. There are 1,878 inhabited households (686 + 1,192 households) for which we do
not have information on the number of household members. For example, these could be cases where a
car was seen parked in the driveway, newspapers were delivered to the household, or lights were on in the
house. Out of the 1,496 households (1,429 +67 households) for which we do have information on the
number of household members, 1,429 (i.e. 95.5 percent) have at least one household member and 67 do
192
not. We assume that the 1,878 households are similar to the 1,429 households where we have reports of
the number of household members. Therefore, out of 1,878 households, we calculate that 1,793
households could have at least one household member. For the 1,429 households with household
members, 928 (65%) have at least one 10- to 25-year-old, so for the 1,793 households we calculate that
1,165 households would contain at least one 10- to 25-year-old. Using the youths/household ratio of 1.52
(Table M-5), we find that 1,165 households would contain approximately 1,771 youths in the 10-25 age
range.
In addition, there are 31 households where there is no information on the presence of 10- to 25year-olds. Out of 1,398 households for whom we do have information on 10- to 25-year-olds, 928 (66.38
percent) have at least one 10- to 25-year-old and 470 have none. Therefore, out of 31 households there
would be 20.5 households with 10- to 25-year-olds and those 20.5 households would contain
approximately 31 youths.
Finally, there are 5 households which contain 10- to 25-year-olds but for which we do not have
the number of youths. If we assume that these households contain youths in the same proportion (1.52
youths per household) as the households for whom we know the number of youths, then these 5
households would contain approximately 8 youths.
Thus, we have a combined estimate of 1,810 additional 10- to 25-year-old youths. So we can
estimate that there were 1,401 to 3,211 missing youths in the 10-25 age range.
6. Summary
The analysis is structured around three main questions:
(1) Why were we unable to get an interview? The main reason for not completing a household
screener (a non-interview) was that the sampling unit was uninhabited, while the second most common
reason was a refusal. When a respondent refused to complete a screener, the Field Interviewers attempted
a “conversion”; these conversions were pursued more aggressively if there was reason to believe an
eligible youth resided in the housing unit.
(2)What type of households were non-interviews? The NIR demographics indicate that the
people who were non-interview respondents look very much like our respondents who completed the
household screener. Of the non-interview respondents, those whom we believe to have eligible children
were more likely to be non-Hispanic white and live in a detached single family housing unit than those
whom we believe not to have eligible youth.
(3) How many household members and youths were missing from our sample? Using NIR data
for 1,429 enumerated households, we estimate 4,289 household members are missing from the NLSY97
sampling universe. Using youth counts for housing units, we count as missed 1,401 youths 10 to 25 years
of age. Household enumerations are reliable since they come primarily from household members who
refused to participate in the household screener but provided sufficient summary information to Field
Interviewers for completing the NIR form. The limited NIR data do not permit us to estimate the number
of these 1,401 youths who would have been eligible for the NLSY97 or PAY97 (ages 12 to 23) and are
thus missing from the study samples. However, we can use estimates of the number of youths counted in
the 17-25 age range (570 youths) to count approximately 831 youths in the 10-16 age range.
We should also keep in mind that there could potentially be eligible youths whom we cannot
count because information on the number of household members or number of youths was not provided
by respondents/informants. In particular, they often gave “don’t know” and “invalid” responses.
Therefore, the actual number of potentially eligible youths could be significantly higher than the number
of youths we were actually able to count from the information provided in the NIR reports. We estimate
193
that there could be at least 1,810 more 10- to 25-year-olds beyond the 1,401 youths counted from the NIR
reports, giving us an estimate that ranges from 1,401 to 3,211 such youths.
194
Figure M-2 Counting the Number of Youths from the Inhabited Households
Inhabited NIR Households
3,374
How many household members?
Don’t Know
1,192
None
470
No household
Have Household
members 67
Members
1,429 households There are
10-25 year olds,
Any 10-25 years old928
s? households
Don’t Know
31
Missing
information
686
Number of 10-25 year olds?
Don’t
Know
5
923
households
1,401 youths
There are
Don’t Know
17-25Any
year17-25
olds year22
olds?
439 households
Number of youths 17-25?
Don’t Know
9
195
Have youths, ages 17-25
430 households,
570 youths
APPENDIX N
Report on NLSY97 Computer Crashes
196
This appendix summarizes the findings from NORC’s research into areas that were possible
causes or contributors to the NLSY97 “hole in the age distribution” problem. The areas we researched
were: incidences of crashed/technical problem cases (codes 72 & 73), cases coded “no housing unit”
(code 83), and the four lowest responding PSUs. Table N-5 lists the blocks where the highest numbers of
crashed/technical problem cases and “no housing unit” cases occurred.
Crashed Cases
We learned that the crashed cases were mainly due to the following:
•
•
•
Low Random Access Memory (RAM)
Reports of insufficient space on the hard drive
Inconsistent battery life
It is important to note that there was no single cause for these crashed cases. All of the categories
listed above have to do with the equipment used for the survey.
Low Response PSUs
The four PSUs with lowest response rates were:
•
•
•
•
PSU 721
PSU 764
PSU 795
PSU 862
31.6 percent
37.4 percent
21.3 percent
59.5 percent .
The following discussion provides the final disposition codes for each case within each of the four PSUs.
The “no action” category reflects households from replicates that were never considered to be part of the
sample, and thus should be ignored.
197
Table N-1 PSU 721
SEG
001
001
001
001
001
001
002
002
002
002
003
003
003
003
003
003
003
003
004
004
004
004
004
004
004
005
005
005
005
005
005
LSTFLDDI
No Action
Complete – Pers
Comp by Paper Sc
Case Crashed
No Address/No HU
Vacant HU
No Action
Complete – Pers
Comp by Paper Sc
Vacant HU
No Action
Complete – Pers
Comp by Paper Sc
Case Crashed
Proxy No 8-28
No Address/No HU
Not Perm Residen
Hostile Refusal
No Action
Complete – Pers
Comp by Paper Sc
Conv Phone Paper
Proxy No 8-28
R Unavail
Refusal
No Action
Complete – Phone
Case Crashed
Not Final inTime
Comp DataUnavail
Vacant HU
Frequency
8
8
7
8
7
10
5
1
11
8
9
16
9
1
1
1
1
1
6
21
1
1
3
2
1
5
1
16
1
1
2
Percent
4.6
4.6
4.0
4.6
4.0
5.8
2.9
0.6
6.4
4.6
5.2
9.2
5.2
0.6
0.6
0.6
0.6
0.6
3.5
12.1
0.6
0.6
1.7
1.2
0.6
2.9
0.6
9.2
0.6
0.6
1.2
Sample Analysis
Original Sample:
173
No Action:
33
Out of Scope:
29
Net Sample:
111
Completes:
80
NIRs:
6
Crash/Tech:
25
Response Rate:
72.1 percent
Percent Crash/Tech:
22.5 percent
Observations
198
Cumulative
Frequency
8
16
23
31
38
48
53
54
65
73
82
98
107
108
109
110
111
112
118
139
140
141
144
146
147
152
153
169
170
171
173
Cumulative
Percent
4.6
9.2
13.3
17.9
22.0
27.7
30.6
31.2
37.6
42.2
47.4
56.6
61.8
62.4
63.0
63.6
64.2
64.7
68.2
80.3
80.9
81.5
83.2
84.4
85.0
87.9
88.4
97.7
98.3
98.8
100.0
In the case of PSU 721, dispositions for all segments within the PSU were given. Segment 005
carries a 10 times greater weight than the others and only had one completed case out of 19 eligible lines.
Of these eligible lines, 16 were crashed/technical problems, one was unavailable, one was incomplete due
to lack of time, and the other was completed. This meant that we had a 5 percent response rate in that
segment which would effect the overall response rate after weighting. It should also be noted that there
were a total of 25 crashed cases in the whole PSU, and 16 of them were in segment 005.
Table N-2 PSU 764
Disposition Category
No Action
Complete – Pers
Complete – Phone
Conv Complete Ph
Comp by Paper Sc
Gate No one 8-28
Case Crashed
Unspec Tech Prob
Proxy No 8-28
Not Final inTime
Inelig. Data
No Address/No HU
Vacant HU
HU Inaccessible
No Ans/NeverHome
R Unavail
Refusal
Frequency
60
21
2
1
30
4
109
1
16
1
1
12
29
2
5
1
4
Percent
20.1
7.0
0.7
0.3
10.0
1.3
36.5
0.3
5.4
0.3
0.3
4.0
9.7
0.7
1.7
0.3
1.3
Sample Analysis
Original Sample:
299
No Action:
60
Out of Scope:
41
Net Sample:
198
Completes:
74
NIRs:
14
Crash/Tech:
110
Response Rate:
37.4 percent
Percent crash/tech:
55.5 percent
199
Frequency
60
81
83
84
114
118
227
228
244
245
246
258
287
289
294
295
299
Percent
20.1
27.1
27.8
28.1
38.1
39.5
75.9
76.3
81.6
81.9
82.3
86.3
96.0
96.7
98.3
98.7
100.0
Observations
There were only 14 NIR cases out of 198 sampled households. Therefore, these could not be
perceived as the cause of the low response rate. Fifty-five point five percent crashed cases could certainly
have affected the response rate and may have been a minimal contributor to the hole in the age
distribution.
Table N-3 PSU 795
Disposition Category
No Action
Complete – Pers
Complete – Phone
Comp by Paper Sc
Gate No one 8-28
Case Crashed
Unspec Tech Prob
Proxy No 8-28
Not Final inTime
Inelig. Data
No Address/No HU
Vacant HU
No Ans/NeverHome
Language Barrier
R Unavail
Refusal
Frequency
66
16
5
6
14
78
66
7
2
5
9
29
11
4
4
7
Percent
20.1
4.9
1.5
1.8
4.3
23.7
20.1
2.1
0.6
1.5
2.7
8.8
3.3
1.2
1.2
2.1
Frequency
66
82
87
93
107
185
251
258
260
265
274
303
314
318
322
329
Percent
20.1
24.9
26.4
28.3
32.5
56.2
76.3
78.4
79.0
80.5
83.3
92.1
95.4
96.7
97.9
100.0
Sample Analysis
Original Sample:
No Action:
Out of Scope:
Net Sample:
Completes:
NIRs:
Crash/Tech:
Response Rate:
Percent Crash/Tech:
329
66
38
225
48
33
144
21.3 percent
64.0 percent
Observations
The number of crashed/technical problem cases amounted to 64 percent of the sampled
households. The NIRs only accounted for approximately 15 percent of the sampled households.
200
Table N-4 PSU 862
Disposition Category
No Action
Complete – Pers
Conv Complete
Comp by Paper Sc
Gate No one 8-28
Case Crashed
Unspec Tech Prob
Proxy No 8-28
Inelig. Data
Demolished
Business
No Address/No HU
Vacant HU
No Ans/NeverHome
Refusal
Frequency
63
57
4
40
7
33
45
19
4
1
1
3
25
4
5
Percent
20.3
18.3
1.3
12.9
2.3
10.6
14.5
6.1
1.3
0.3
0.3
1.0
8.0
1.3
1.6
Frequency
63
120
124
164
171
204
249
268
272
273
274
277
302
306
311
Percent
20.3
38.6
39.9
52.7
55.0
65.6
80.1
86.2
87.5
87.8
88.1
89.1
97.1
98.4
100.0
Sample Analysis
Original Sample:
No Action:
Out of Scope:
Net Sample:
Completes:
NIRs:
Crash/Tech:
Response Rate:
Percent crash/tech:
311
63
30
218
127
13
78
58.3 percent
35.8 percent
Observations
Judging by how few NIRs there were in this PSU, the crashed cases (just over one-third of the
sampled households) caused the biggest problem. If we had been able to resurrect the crashed/technical
problem cases, we probably would have had a much higher response rate.
No Housing Units
There were a total of 1,580 cases that were coded 83 (No Housing Unit). This is an average of
7.9 per PSU. In our research, we attempted to look for clusters of these that would identify a major
problem with the listing. We identified three potential problems:
•
•
•
One year’s time between the listing and the screening, which might explain why certain
neighborhoods had changed;
Demolished houses coded as “no housing unit” because the interviewer had no way of knowing that
there were ever houses there; and
“Listing errors,” such as back doors to apartments being listed as separate units.
Conclusions
201
The large number of crashed/technical problems could have contributed to the “hole in the age
distribution” problem, although this study reveals no definitive information about issue. Nationwide there
were, in total, 1,318 crashed cases. By subtracting 357 cases (the total from the four lowest response rate
PSUs), we are left with 961 across the remaining 196 PSUs, an average of 4.9 per PSU.
As far as the “no housing unit” cases, it is highly unlikely that decentralized occurrences of these
cases had much effect on the hole in the age distribution.
202
Table N-5 Location of Largest Numbers of Crashed/Technical Problems and “No HU” Cases
PSU
SEG
BLK
708
738
738
739
739
739
742
742
742
778
792
792
796
796
796
796
821
821
821
821
821
821
821
826
854
854
854
001
016
023
001
003
014
006
007
008
008
003
024
002
003
003
010
003
004
004
005
008
008
009
005
009
009
009
110
101
201
111
502B
142
710
705
410
601
110
303
208
101
401
117
512
204
205
110
208
209
401
316
202
203
204
83: No
HU
72:
Crash
73: Tech
Problem
12
9
10
10
9
26
10
14
11
8
9
8
11
14
9
16
13
1
12
9
17
12
13
14
10
1
10
11
7
20
COMMENTS
FI: U of VT was replacing dormitories; these buildings used for student housing
No explanation
Field manager does not recall this number of crashed cases.
May have been college housing, in transition between listing date and screening
May have been college housing, in transition between listing date and screening
Trailer park had to be relisted
Mostly shells of houses; no roofs; no windows; homeless men on street
Field manager has no notes in her log that any cases crashed in this segment
Field manager says only 1 case crashed here; does not recall details
Field manager says locked gate community; could get in only at certain times of day
Field manager deceased; no information available
Field manager deceased; no information available
Field manager describes technical and personnel difficulties. Technical: difficulties with the software interface and
Performance of laptops in use by field staff. Often
Unable to transmit cases electronically, had to transmit manually
Suggest we talk further with this field manager
See notes above for this field manager.
See notes above for this field manager.
See notes above for this field manager.
See notes above for this field manager.
See notes above for this field manager.
See notes above for this field manager.
See notes above for this field manager.
Wooded rural area; hunting; vacation cabins; trailer park w/vacant lots; unsafe
High producing FI was given new Think Pad. When loaded, @ 200 cases crashed.
Field manager is SURE majority of cases eventually resolved with more conventional disposition.
FIELD MANAGER is SURE majority of cases eventually resolved with more conventional disposition.
Each PSU-SEGMENT-BLOCK combination containing a group of at least 8 cases with a disposition of 72 (case crashed/data unavailable) and/or 73 (Data unavailable/Unspecified technical
problem OR a disposition of 83 (No such address, no housing unit) is listed. An attempt was made to contact the field manager who supervised the NLSY97 screening/sampling task for that
location and discuss reasons for these dispositions.
203
Appendix O
Report on NLSY97 Debriefing Interviews
204
In 1997 NORC screened approximately 90,000 U.S. households to find eligible youths for
NLSY97/PAY97 in 1997. After the screening was completed it was found that the number of people in
the eligible age range was considerably lower than expected. Debriefing interviews were conducted with
a sample of field interviewers (FIs) to explore how two factors may have contributed to this shortfall in
the sample. The first factor is the technical difficulties that FIs experienced with the laptops provided for
screening. Computers were reported to have “crashed” often during screening interviews, perhaps most
often in households with eligible youths, when the screening process was the lengthiest. This would have
provided opportunities for eligible youths to be omitted from the sample. The second factor that may
have contributed to the shortfall in the sample is the informants’ knowledge of the age range of eligibility
for NLSY97/PAY97. A brochure about NLSY97 that was sent to selected households stated explicitly
that youths ages 12 through 17 were eligible for the study. In addition, FIs were informed of the eligible
age range for both NLSY97 and PAY97. Thus, informants may have learned in advance of the screening
the ages of eligibility, allowing them to avoid participation in the study by misreporting on the household
roster. Patterns in the age frequency data, in which the shortfall corresponds well to the exact ages that
are eligible for the study, suggest that informants may have misreported.
This report presents findings from debriefing interviews conducted with selected field
interviewers who participated in the NLSY97/PAY97 screening in 1997. The debriefing questionnaire
addresses both the computer problems that FIs encountered and possible misreporting by informants.
Survey Procedures
Questionnaire. A copy of the questionnaire appears in Attachment 1.
Respondents. The respondents were 96 FIs who screened households for NLSY97/PAY97 in
1997. These FIs were selected at random from 432 FIs who conducted least 10 screening interviews and
were still on NORCs list of active interviewers as of October 12, 1998.22 The 96 FIs worked on 24.0
percent of the 65,530 completed screening interviews and 24.7 percent of the 90,924 total screeners (that
is, including non-interview reports and other cases).
The FIs were asked about the number of years they had been interviewing for NORC and whether
they had any prior experience with CAPI projects (Questions 11 and 12). FIs ranged from one year to 20
years of employment at NORC. Of the 96 FIs, 68.8 percent reported having worked on CAPI projects
before the NLSY97 screening.
Procedure. Six Field Managers (FMs) who served on NLSY97 Round 1 were recruited to
conduct debriefing interviews with the sampled FIs. Each FM conducted sixteen interviews by telephone.
Interviews were conducted between October 21 and 31, 1998.
22
Data on FIs who worked on NLSY and the final disposition codes of the screeners they worked on is
based on lists created at the close of the field period in September/October of 1997. Since these data were
subsequently checked and further screeners were completed in 1998, the lists used to create the sample
frame may contain some omissions and inaccuracies.
205
Main Findings
Computer problems. Seventeen FIs stated that they conducted only paper-and-pencil (PAPI)
screeners (Question 1). Of the 79 FIs who did use the computer to screen, 32 (40.5 percent) reported
experiencing problems with their computers (Question 2). As Table O-1 shows, computer problems
prevented the completion of a screening interview in progress five or fewer times for 30 of the FIs; one FI
reported experiencing problems on 20 occasions and one other FI reported 100 such problems (Question
3). Without comparison data from other CAPI studies it is not possible to judge how unusual this is.
Most of the FIs who experienced problems (n=26, 78.8 percent) reported that the computer crashes
occurred at least once at a household where they thought there were eligible youth (Question 4).
Although it is assumed that FIs later attempted to complete screening for all such crashed cases, it is not
clear what their success rate was in securing cooperation again upon returning to the household.
FIs were asked to give a brief description of their computer problems (Question 5). These openended descriptions were then categorized (see Table O-2). As Table O-2 indicates, the most commonly
reported problems were that the computer simply “crashed “ or “froze” during the screening interview.
As Table O-3 shows, FIs without prior CAPI experience were somewhat more likely to experience
problems. Of those FIs who did CAPI screening (n=77), the 49 FIs with prior CAPI experience had
fewer problems with their computers (32.7 percent with problems vs. 67.3 percent without) than the 28
who did not have prior CAPI experience (57.1 percent with problems vs. 42.9 percent without problems).
Misreporting by the informant. Households that wanted to avoid participating in the
NLSY97/PAY97 studies could do so in a number of ways. They could refuse to participate in a screening
interview, or, if they did participate and were selected for the main study, they could refuse to do the main
interview. However, since refusing to be interviewed can lead to subsequent visits by interviewers
attempting to secure cooperation, an alternate way of refusing to participate is to misreport household
information. Informants could have discovered the eligible age range through several sources. The
NLSY97 brochure mailed to households stated the exact ages of eligibility for this study. Neighbors who
were already screened could supply the relevant information. In addition, the FIs themselves could have
told informants. In the debriefing interview FIs were asked what they told informants about ages of
eligibility for the NLSY97 study prior to the actual screening interview (Question 9). FI’s open-ended
responses to this question were coded into categories (see Table O-4). Thirty-eight FIs (39.6 percent)
indicated that they told responses about the ages of eligibility; twenty-six of these stated specific ages in
their responses to the debriefing items. It appears from this finding that many FIs did tell informants the
age range of interest. It is possible to speculate that some informants tailored their responses to avoid
participating in the survey. However, no data are available that can indicate how often this may have
happened.
If informants were misreporting, it was not often obvious to the FIs. According to FI’s
judgments, informants seemed cooperative and truthful in their responses to the screener. On a 1 to 5
scale, with 1 representing “very easy” and 5 representing “very difficult”, the FIs mean rating of the
difficulty of gaining cooperation was 2.6 (Table O-5). Although respondents could have learned some
information about the eligible ages for the NLSY97 and used this information to misreport household
members, the FIs did not feel that this was the case. FIs judged that informants rarely read the advance
letter and brochure (which stated the eligible ages for NLSY97) and most felt that informants were
reporting accurately on the household roster (Tables O-6 and O-7). FIs were also asked to list the most
common questions or concerns that informants expressed about NLSY97. As can be seen in Table O-8,
FIs were asked typical questions about confidentiality, background of the study, and how long the
interview would take.
The types of questions informants asked were not unusual and did not suggest any particular reasons why
206
they would try to avoid participating in NLSY97.
Conclusions
Although many FIs reported experiencing problems with their laptop computers, the frequency of
these problems was not high. Therefore, computer difficulties alone cannot explain the missing youth in
the sample. Many FIs did tell informants about the age ranges of eligibility, possibly allowing for some
misreporting on the household roster. However, most FIs did not get the impression that misreporting
happened very often. Rather, most FIs thought informants were cooperative and truthful in completing
the screener.
Table O-1 Number of Times a Computer Problem Prevented
Completion of a Screening Interview
Number of times computer
problem experienced.....
1
Number of FIs reporting
9
2
7
3
7
4
5
5
2
20
1
100
1
Table O-2 Types of Computer Problems Experienced by FIs
Computer problem
Number of reports
Computer crashed, screen locked or frozen
28
Computer slow
5
Battery dead
3
Cannot access case or case not spawned
5
Problem with skips
2
Other technical problem
6
Other non-technical problem
8
Note: This table includes 57 problems reported by 36 FIs.
207
Table O-3 Computer Problems in NLSY97 and Prior CAPI Experience
NLSY97 computer problems?
Prior CAPI
experience?
Problems
No problems
Totals
Yes
16
33
(49)
No
16
12
(28)
(32)
(45)
(77)
Totals
Table O-4 What FIs Told Informants About Who is Eligible to Participate in NLSY97
n
(%)
22
(22.9%)
11
(11.5%)
38
(39.6%)
8
(8.3%)
17
(17.7%)
Followed protocol or script
Scientific sample, computer selects Rs
Eligible ages
Other
FI does not remember
Table O-5 Number of FIs Giving Each Rating of Difficulty in Gaining Cooperation
Rating
n
(%)
1
very easy
7
(7.3%)
2
3
4
40
(41.7%)
40
(41.7%)
6
(6.3%)
mean rating of 2.6
n = 96
208
5
very difficult
3
(3.1%)
Table O-6 FIs’ Ratings on How Often Informants Read Advance Letter and Brochure
Rating
1
very often
2
3
4
4
(4.3%)
7
(7.4%)
22
(23.4%)
30
(31.9%)
N
(%)
5
not often
at all
31
(33.0%)
mean rating of 3.8
n = 94
Table O-7 FIs’ Ratings on How Often Informants Did Not Report Accurately on Household Roster
Rating
1
very often
2
3
4
1
(1.0%)
2
(2.1%)
4
(4.2%)
32
(33.3%)
5
not often
at all
57
(59.4%)
mean rating of 4.4
n = 96
Table O-8 FI Reports of Questions and Concerns Expressed by NLSY97 Informants
Question/concern
Number of reports
Confidentiality
45
How were we selected?
21
What kinds of questions will be asked?
10
Who is conducting study and how will this information be used?
31
How long will this take?
29
Why should I do this?
6
Is this voluntary?
10
Payment
1
Military/FBI/police/IRS
6
Government
5
General distrust
6
Miscellaneous
25
Note:
This table includes 195 questions and concerns reported by 86 FIs.
209
Attachment 1
NLSY97 Interviewer Debriefing Questionnaire
October 1998
This debriefing interview conducted by:
Date:
/
/
FI Name _______________________________________________________ _
Phone # _________________________________________________________
Respondent #
(Also write respondent number on top of next page.)
WHEN YOU HAVE COMPLETED THIS INTERVIEW PLEASE DETACH AND KEEP THIS PAGE.
RETURN REMAINING QUESTIONNAIRE PAGES TO....
LISA LEE
55 EAST MONROE, 48TH FLOOR
CHICAGO, IL 60603
THANK YOU!
210
Respondent # _______
(copy from face page)
NLSY97 Field Interviewer Debriefing Questionnaire
October 1998
We are interested in learning about field interviewers’ experiences in conducting screening interviews for
the NLSY97 in 1997. Therefore we are asking a number of interviewers to complete this debriefing
questionnaire. Your participation is voluntary, but we hope you will take the time to provide us with your
feedback. The information you provide will allow us to improve our procedures for future studies.
Participants will be identified by number only, not by name, so your name will not be associated with any
of your answers.
1.
Were you a PAPI-only screener?
Yes……………………………………. 1 (GO TO Q. 6)
No …………………………………….. 2 (GO TO Q. 2)
2. Some interviewers had difficulty with the computers provided for completing screening interviews.
We are interested in learning about times when a computer problem made it necessary to discontinue
an interview and return to the household at a later time to complete it. Did this ever happen to you?
Yes……………………………………. 1 (GO TO Q.3)
No …………………………………….. 2 (GO TO Q. 6)
3.
About how many times did this happen?
Times
4.
Did this ever happen during a screening interview at a household where you thought there were
young people eligible for the study?
Yes……………………………………. 1
No …………………………………….. 2
211
5.
6.
Please describe briefly the problems you experienced with your computer.
1.
______________________________________________________________________
2.
______________________________________________________________________
3.
______________________________________________________________________
4.
______________________________________________________________________
A. In general, how easy or difficult was it to get informants’ cooperation in completing the
screener? Please give a rating from 1 to 5, with 1 being “very easy” and 5 being “very difficult.”
Very easy
1
2
3
4
Very difficult
5
B. What makes you give this answer? ______________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
7.
How often did informants seem to have read the advance letter and brochure that NORC sent?
Please give a rating from 1 to 5, with 1 being “very often” and 5 being “not often at all.”
Very often
1
2
3
4
212
Not often at all
5
8.
A. How often did you feel that informants were not reporting accurately on the household roster?
Please give a rating from 1 to 5, with 1 being “very often” and 5 being “not often at all.”
Very often
1
2
3
4
Not often at all
5
B. What makes you give this answer? ______________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
9.
Prior to the actual interview, what did you tell informants about who was eligible to
participate in the NLSY97 study?
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
213
10.
What were the three most common questions or concerns informants had about NLSY97?
LIST EACH QUESTION BELOW AND ASK:
How did you address this question/concern?
1.
Question: _______________________________________________________________
_______________________________________________________________________
Answer: ________________________________________________________________
_______________________________________________________________________
2.
Question: _______________________________________________________________
_______________________________________________________________________
Answer: ________________________________________________________________
_______________________________________________________________________
3.
Question: _______________________________________________________________
_______________________________________________________________________
Answer: ________________________________________________________________
_______________________________________________________________________
11.
How many years have you been an interviewer for NORC?
Years
12.
Had you worked on any CAPI projects before the NLSY97 screening?
Yes……………………………………………… 1
No ……………………………………………… 2
This is the end of the interview. Thank you for your help!
214
APPENDIX P
Analysis of Interview Validation
215
Project Summary
In view of the NLSY97 Round I “observed” undercount, an analysis of the validation interviews
was completed. The two main objectives of the analyses were: (1) to evaluate the performance of the
interviewers using the respondents’ comments; and (2) to analyze the validation process of NORC’s field
staff for the NLSY97 data collection efforts. This report outlines the data collection efforts, describes the
analyses performed on the data gathered, and discusses the results.
Data Collection
At the end of January 1997, household screeners and interviews were fielded for NLSY97, and
the fieldwork continued until November of 1997. The validation process of the screeners and interviews
began in March of 1997. NLSY97 validation specifications included: (1) validation of one person only in
each selected household; (2) validation of 0.5 percent of all screeners completed in households that did
not contain any eligible sample members; (3) validation of 4.0 percent of parent completions per
interviewer (each interviewer needed a minimum of one complete parent case validated); and (4)
validation of 5 percent of youth completions per interviewer (each interviewer needed a minimum of two
completed youth cases validated).
Initially the field managers (FMs) conducted validation interviews at their own discretion; in May
of 1997 the validation process was formalized and carried out by a group of NORC interviewers.
According to the validation report, a total of 1,390 validation interviews were conducted, including 476
screeners, 518 youth, and 396 parent interviews. The final count of the validation interviews that were
electronically entered and used for analysis was 1,296, including 347 screeners and 949 youth and parent
interviews combined. The difference between 1,390 and 1,296 (94 interviews) can be explained by the
fact that some of the initial validation interviews conducted by the FMs themselves may not have been
entered in the formal validation questionnaires.
Data Analysis
Among the 1,296 validation interviews for which there was an explicit validation questionnaire,
347 were screeners and 949 were parent and youth interviews. For these cases, the parent and youth
interviews are not distinguished from each other as it is not possible to do so from the data available in
the validation questionnaires alone. A link file exists in the archives which would require additional work
to retrieve and process.
Validation interviews were conducted on cases which were originally interviewed between
January and October of 1997, while the validation interviews were conducted between March and
November of 1997. Overall, the validation status of the 1,296 cases is as follows: 90.2 percent (1,169)
were successfully validated, 7.0 percent (91) did not remember, 1.5 percent (20) were refusals, 0.2
percent (2) were break-offs, and 1.1 percent (14) were blank. For the ineligible screeners, only 69.7
percent (242) were successfully validated, 24.8 percent (86) did not remember, 4.6 percent (16) refused,
0.6 percent (2) were break-offs, and 0.3 percent (1) were undetermined. The successful validation status
for the parent and youth interviews was an overwhelming 97.7 percent (927), 0.5 percent (5) did not
216
remember, 0.4 percent (4) were refusals, and 1.4 percent (13) remained undetermined. (A paper and
pencil follow-up was conducted for all unsuccessful validations, but it is possible that the final disposition
may not have been entered in the computer system.)
A cross tabulation of interview dates and validation interview dates showed that, for the ineligible
household screeners, and the youth and parent interviews, the majority of the validation interviews had
occurred at least one month after the original interview. A separate analysis, consisting only of those 105
ineligible household screener cases which were not successfully validated, revealed that only 15.2 percent
(17) of the respondents were validated within two months of the original interview, and 28.5 percent (30)
of the respondents were validated at least six months after the original interview.
Over four-fifths (1,076) of the respondents reported that the initial interview had occurred in
person, 3.5 percent (46) by telephone, and 13.1 percent (170) did not respond to this question. The
majority of those who did not answer the question regarding the mode of the interview were screener
cases (154). Among the parent and youth interviews, 93.9 percent (891) reported having completed the
initial interview in person.
In the comments section, 338 respondents out of 1,296 for the validation study made comments
regarding the interviewers’ mannerisms. In total, 21 percent (272) reported in the affirmative that the
interviewer was polite and courteous, 3 percent (41) made comments which were categorized as negative,
2 percent (25) made comments which were categorized as neutral, and 74 percent (958) made no
comments. The majority of the respondents to the youth and parent interview who made negative
comments complained about the length of the interview and/or of not receiving the promised
compensation for the youth test. Ninety-five percent (898) of the parent and youth respondents reported
getting paid for the interviews. The payment varied between five and ninety-five dollars.
Discussion
Overall, the validation interview analysis revealed one major lapse on the part of the field staff in
not conducting and completing the validation interviews within a set time period in order to prevent
problems with memory recall on part of the respondents. The ineligible household screener consisted
only of a few questions regarding the composition of the household, and that brief meeting could easily
be forgotten by the respondents. In addition, identical questionnaires were used to validate both the
screener and the youth and parent interviews, and thus do not validate any particular responses to any of
these components. A separate validation interview questionnaire should have been designed to ask the
respondent to provide information regarding the age composition of the household; this could then have
been matched to the original response during the screener.
217
Conclusion
The analyses of the validation interviews failed to reveal any poor conduct on part of the
interviewers, as most of the comments of the respondents were positive and the few negative comments
were beyond the control of the interviewers. On the other hand, the field staff failed to conduct timely
validation interviews, especially for the ineligible household screeners, which leave our results somewhat
inconclusive.
Table P-1 NLSY97 Round 1 Validation Interviews
Type
Screener
Successful
Failed
Other
57.8%
20.2%
22%
(275)
(96)
(105)
Youth
98.6%
0%
1.4%
(511)
(0)
(7)
Parent
97%
0.5%
2.5%
(384)
(2)
(10)
All
84.2%
7%
8.8%
(1,170)
(98)
(122)
Source: Based upon the report of validation interviews, not upon the validation forms
themselves.
Table P-2 NLSY97 Round 1 Validation Interviews
Ineligible Screeners Only
Validation Status
Successful
Other than Successful
Don’t remember
Refusal
Break off
Unknown
Total
Number
242
105
86
16
2
1
347
218
Percent
69.7
30.3
24.8
4.6
0.6
0.3
100.0
Total
100%
(476)
100%
(518)
100%
(396)
100%
(1,390)
Table P-3 NLSY97 Round 1 Validation Interview
Respondent Comments Regarding the Survey or Experience with Interviewer
Type of Comment
Positive
Screener
10%
(37)
2%
(6)
2%
(7)
86%
(297)
100%
(347)
Negative
Neutral
No comment
Total
Youth or Parent
25%
(235)
4%
(35)
2%
(18)
69%
(661)
100%
(949)
Total
21%
(272)
2%
(41)
2%
(25)
74%
(958)
100%
(1,296)
Table P-4 Elapsed Time Between Screener and Validation Interview for 105 Screeners for
Which Validation was Other Than Successful
Month of
Screener
February
March
April
May
June
July
August
September
Total
Number Of Months Between Screener and Validation Interview
0
3
1
4
1
1
2
4
5
1
13
2
1
3
3
7
3
1
14
5
20
4
6
3
3
4
16
219
5
2
1
2
3
8
6
2
2
3
7
7
3
12
15
8
14
14
9
1
1
APPENDIX Q
Analysis of Parent Education
for the NLSY97
220
An analysis of parent education was conducted as part of NORC’s exploration into differences
between NLSY97 youths and the national universe of 12- to 16-year-olds.23 The highest grade attained
by the mother and father are examined in this appendix, along with a summary of the data set, which
identifies numbers of youths living in households with one or both parents or living without either parent.
For comparison, the same analyses are shown for families with resident youths aged 12 to 16 responding
to the March 1997 Current Population Survey (CPS)24.
Framework for Education Analyses. The analyses that follow were conducted at the youth
level. Highest grade attained was captured for the parents of all eligible NLSY97 youths (9,022 total).
For the CPS, all 12- to 16-year-old youths were selected for analysis (9,130 total), and the highest grade
attained by the parents of those youths was extracted. For both the NLSY97 and the CPS, only resident
parents were used in the analyses; any parents not living with the youths were excluded.
Education is broken down into seven distinct categories: 8th grade or below, 9th through 11th
grades, 12th grade (which may or may not include a high school diploma), some college (no bachelor’s
degree), a BA or a BS degree, and graduate level coursework or a higher degree (including master’s
degrees, professional degrees, and doctorate degrees). Youths for whom parents were in ungraded
schooling, did not know his or her highest grade, or refused to provide education information were left
out of the education comparison; the extent of missingness in the NLSY97 and CPS data is discussed
later.
Both the NLSY97 and the CPS analyses were conducted on an unweighted basis. Raw counts
and percentages associated with those counts are displayed in each of the tables shown below.
All of the tables showing comparisons between CPS and NLSY97 display the NLSY97 overall
and by sample type. Because the supplemental (SU) sample consists only of minority youths, educational
attainment for parents of this portion of the NLSY97 will differ quite noticeably from the cross-sectional
(CX) sample, which was designed to be representative of the entire population of 12- to 16-year-olds.
Thus, comparisons between the CPS and NLSY97 should be based mainly on the CX portion of the
NLSY97 sample.
Differences in Question Wording. While both the CPS and NLSY gather information about
parents’ highest grade attained, the question wording differs quite widely between the two surveys. The
CPS asks for educational attainment for adults (persons aged 16 and older) grouped into the following 16
categories: less than first grade; 1st, 2nd, 3rd, or 4th grade; 5th or 6th grade; 7th or 8th grade; 9th grade; 10th
grade; 11th grade; 12th grade (no diploma); high school graduate (diploma or equivalent); some college but
no degree; associate degree in college (occupation/vocation program); associate degree in college
(academic program); bachelor degree; master degree; professional school degree (e.g., MD, DDS, LLB,
JD); and doctorate degree (e.g., PhD, EdD). The NLSY grouped its parental education variable into 24
categories based on year of schooling: did not attend; pre-kindergarten; kindergarten; 1st grade; 2nd grade;
23
Final updating of the NLSY97 Round 1 disposition codes was done in November 1999, in light of all evidence on
eligibility collected to that point in time, including certain evidence collected during Round 2 operations. Because
the tabulations cited in this appendix reflect an earlier generation of the Round 1 disposition codes, there may be
small discrepancies between counts cited here and the final counts cited in the main body of this NLSY97 Technical
Sampling Report.
24
The CPS Hispanic supplement was not included in these analyses.
221
3rd grade; 4th grade; … ; 9th grade; 10th grade; 11th grade; 12th grade; 1st year of college; 2nd year of
college; … ; 7th year of college; 8th year of college; and ungraded. These variable value differences
between the two surveys necessitated the creation of a final grade variable, which combined and matched
grade levels from the CPS and NLSY as comparably as possible. Table Q-1 helps to illustrate these
classifications by showing which of the original categories from the two surveys is included in each of the
final categories.
Table Q-1 CPS and NLSY97 Components of Final Grade Categories
Final Grade Category
th
8 Grade or Below
9th through 11th Grade
12th Grade/High School Diploma
Some College, no BA/BS
BA/BS
Graduate Coursework/
Higher Degree
CPS Components
Less than first grade through 7th or 8th
grade
9th grade through 11th grade
12th grade (no diploma), h.s. graduate
Some college but no degree through
associate degree (academic program)
Bachelor’s degree
Master’s degree, professional degree,
doctorate degree
NLSY97 Components
Did not attend through 8th grade
9th grade through 11th grade
12th grade
1st year of college through 3rd
year of college
4th year of college
5th year of college through 8th
year of college
This matching of CPS and NLSY grade components is far from perfect, but it is probably the best one can
do based upon the component categories presented in the CPS and NLSY questions.
Comparison of the Data Sets. Table Q-2 displays the CPS and NLSY97 data sets, broken down
by the parent(s) with whom the youths reside. Included among the categories are youths living with both
parents, youths living with their father only, youths living with their mother only, and youths living with
neither parent. As shown here, exactly the same percentage of CPS and NLSY CX youths (68.4 percent)
were in a household with both parents. Similar patterns exist for other living situations. For the CPS, 4.5
percent of youths live with their father only, compared to 3.7 percent for the NLSY CX sample, and 22.7
percent of CPS youths lived with their mother only, versus 24.2 percent for NLSY CX youths. Both data
sets had a small percentage of youths living with neither parent (4.4 percent for CPS, 3.7 percent for
NLSY).
Notice that less than half (46.6 percent) of NLSY SU youths live with both parents, more than 20
percent fewer than for the CPS and NLSY CX samples. Further, a much larger percentage live with their
mother only (42.9 percent for NLSY SU, compared to 24.2 percent for NLSY CX) and with no resident
parents at all (7.3 percent, versus 3.7 percent for NLSY CX). This illustrates the vast differences between
the CX and SU sample composition, and their potential effects on other comparisons.
222
Table Q-2 Youths by Presence of Parent—Comparison of Data Sets
CPS
N
Data Set Breakout
%
NLSY--CX
N
%
NLSY--SU
NLSY--Total
N
%
N
%
Youth living with both parents
Youths living with father only
Youths living with mother only
Youths with no resident parents
6,242
412
2,076
400
68.4
4.5
22.7
4.4
4,630
251
1,640
249
68.4
3.7
24.2
3.7
1,050
72
966
164
46.6
3.2
42.9
7.3
5,680
323
2,606
413
63.0
3.6
28.9
4.6
TOTAL
9,130
100.0
6,770
100.0
2,252
100.0
9,022
100.0
Missing Data. Table Q-2 shows that approximately 4 percent of youths in both the CPS and
NLSY97 CX samples do not live with either parent. For those youths, no parent education data will be
available, and they will be left out of the analyses completely. Table Q-3 displays the level of missing
data for the highest grade attained variables in the CPS and NLSY97 data sets given that a mother or
father is present in the youth’s household. In other words, for the youths with resident parents, the
amount of missing data for each parent education variable will be presented. For the CPS, missing data
signifies cases for which a variable value was imputed. The NLSY97 groups three categories as missing:
parents who indicated ungraded schooling as the highest grade attained, those who did not know his or
her highest grade, and those who refused to provide education information. As shown in Table Q-3, the
CPS had less than one percent missing in each of the education variables analyzed here, while the
NLSY97 CX sample is missing between 3.3 and 3.9 percent of the data for these variables.
Table Q-3 Extent of Missingness of the Data
Variable
Father’s Highest Grade Attained
Mother’s Highest Grade Attained
CPS
N
%
52
56
0.8
0.7
NLSY--CX
N
%
189
206
3.9
3.3
NLSY--SU
N
%
114
85
10.2
4.2
NLSY--Total
N
%
303
291
5.0
3.5
In the analyses that follow, imputed values are included for the CPS. In the case of the NLSY97,
the percentages displayed are based on complete data only. Therefore, for the NLSY97, we have
implicitly assumed that the data were missing at random. In other words, we attribute the grade
distribution of the completed cases to the missing cases.
Analysis of Highest Grade Attained. Table Q-4 compares the CPS to the NLSY97 based on
father’s highest grade attained. Again, comparisons should be made mainly between the CPS and the
NLSY97 CX samples. As seen below, the CPS youths have slightly fewer fathers with an education
below twelfth grade (13.4 percent, compared to 16.8 percent for NLSY97), the same percentage with a
twelfth grade education (33.4 percent for CPS, 33.3 percent for NLSY97), and slightly more with at least
some college (53.1 percent for CPS compared to 49.8 percent for NLSY97). However, category by
category, the differences are quite small. Also, it is interesting to note that, while fewer NLSY97 youths
had fathers with college-level coursework at or below a bachelor’s degree, more of their fathers had
graduate-level coursework or a higher degree.
Table Q-4 Father’s Highest Grade Attained
223
Highest Grade Attained
CPS
N
NLSY--CX
NLSY--SU
NLSY--Total
%
N
%
N
%
N
%
8th Grade or below
9th through 11th Grade
12th Grade / High School Diploma
Some College, no BA / BS
BA / BS
Graduate Coursework / Higher Degree
388
507
2,224
1,764
1,052
719
5.8
7.6
33.4
26.5
15.8
10.8
288
503
1,563
1,053
700
585
6.1
10.7
33.3
22.4
14.9
12.5
237
154
350
171
70
26
23.5
15.3
34.7
17.0
6.9
2.6
525
657
1,913
1,224
770
611
9.2
11.5
33.6
21.5
13.5
10.7
TOTAL
6,654
100.0
4,692
100.0
1,008
100.0
5,700
100.0
Table Q-5 shows the same comparison for mother’s highest grade attained. As seen here, similar
patterns exist for mothers as did for fathers. Again, while fewer of the CPS youths had mothers with less
than a twelfth grade education compared to the NLSY CX sample (13.4 percent for CPS versus 17.6
percent for NLSY), and more of the CPS youths had mothers with a twelfth grade education through a
bachelor’s degree (81.2 percent for CPS compared to 74.7 for NLSY), the individual category differences
are modest. And, once again, the NLSY had more youths with mothers who went beyond a bachelor’s
degree (7.7 percent for NLSY, compared to 5.3 percent for CPS).
Table Q-5 Mother’s Highest Grade Attained
Highest Grade Attained
CPS
N
%
NLSY--CX
N
%
NLSY--SU
N
%
NLSY--Total
N
%
8th Grade or below
9th through 11th Grade
12th Grade / High School Diploma
Some College, no BA / BS
BA / BS
Graduate Coursework / Higher Degree
461
659
3,137
2,407
1,214
440
5.5
7.9
37.7
28.9
14.6
5.3
338
729
2,194
1,534
801
468
5.6
12.0
36.2
25.3
13.2
7.7
353
416
672
335
115
40
18.3
21.5
34.8
17.3
6.0
2.1
691
1,145
2,866
1,869
916
508
8.6
14.3
35.8
23.4
11.5
6.4
TOTAL
8,318
100.0
6,064
100.0
1,931
100.0
7,995
100.0
The biggest differences for both father’s and mother’s highest grade attained occur at the 9th
through 11th grade and some college, no BA/BS levels. While the definitions for 9th through 11th grades
are identical for the CPS and the NLSY, there are distinct differences for the college years. The CPS
categorizes persons with any amount of college below a bachelor’s degree (including associate’s degrees)
into the some college category, while the NLSY, which only collects year of school but not degree, only
includes persons with one, two, or three years of college in this category. This anomaly carries over into
the later college categories, also. For instance, the CPS places only persons who attain a bachelor’s
degree into the BA/BS category, while the NLSY places all persons with four years of college into this
group. Thus, any of these parents who attended college for four years but did not obtain a bachelor’s
degree at all would be incorrectly included in the “BA/BS” category. Furthermore, those parents who did
224
receive a bachelor’s degree, but did not obtain it until his or her fifth year of college or later, should be
included in this category but would not be. The same applies to the final category, graduate coursework
or higher degree. The CPS only includes those persons who have actually attained a master’s,
professional, or doctorate degree into this category. For NLSY, all persons whose highest grade attained
was five or more years of college were included here. In other words, a person who had completed a
bachelor’s degree after six years would be incorrectly included here, as would someone who attended
college for five years, for example, and did not receive any degree at all. Therefore, the higher percentage
of NLSY youths with parents who attained more than a bachelor’s education may be explained by these
definitional differences. Rather than being placed in this category, perhaps the surplus would be spread
across the previous two college-level categories, helping to minimize the discrepancies there. However,
despite the definitional differences that do exist, the grade distributions presented here are surprisingly
similar.
While we could produce tests of equivalence of the CPS and NLSY97 sample distributions, we
would not believe such tests to be appropriate because of the definitional differences between the surveys
and their implications discussed above. Nevertheless, we performed a simple t-test to compare the means
of the weighted distributions of father’s and mother’s highest grade attained for the two samples. The
fathers of NLSY97 youths have an average highest grade attained of 3.5 out of the 6 total categories
(somewhere between 12th grade and some college, no BA), while fathers of CPS youths have a mean
education of 3.7 on the 6-point scale. Likewise, mothers of NLSY97 youths attained an average highest
grade of about 3.3 out of 6, compared to 3.5 out of 6 for the mothers of CPS youths. The t-values
associated with the differences in means are 4.9 and 5.5, respectively, which are both significant at the
0.01 level; still, the variances of these distributions were found to be not significantly different. However,
besides the definitional and procedural differences between the surveys discussed earlier, the criteria for a
formal t-test, including simple random sampling, are not met here. At this writing we have not yet
estimated the design effects for the NLSY97 sample, but are certain that they will be greater than one.
Borrowing the design effect for the full NLSY79 sample of youths (DEFF approximately equal to 2), the
results of the t-tests would be smaller, but would still remain significant. Finally, we must note that these
samples, when weighted, are enormous, and with any sample large enough many differences appear to be
significant. Taken as a whole, these limitations further convince us of the inadequacy and
inappropriateness of this type of this test.
Summary. Given sampling variability and definitional and procedural differences
between CPS and NLSY parent grade distributions, the comparisons are reassuring. Our analyses do not
prove that there is no difference between NLSY and CPS youths with respect to parents’ educational
attainment. Yet, in our view, these data offer no conclusive evidence of important differences. On
balance, the analyses reveal little or no evidence of nonrepresentativeness of the NLSY sample.
225
APPENDIX R
Two Comparisons of Expected
vs. Observed Youth Counts
226
In this appendix, we compare actual NLSY97 youth eligible counts with estimated expected
counts in two ways.25 First, we compare these counts at the sample (CX vs. SU) level, paying particular
attention to the two sampling strata (HIGH vs. LOW-LOW). Second, we compare these counts at the
PSU level, looking for patterns and relationships in the shortfalls.
In order to make these comparisons, we needed to estimate the expected counts of NLSY97 youth
eligibles that we would find at the segment level. These expected segment counts are described in this
paragraph. For each and every selected segment, we obtained 1990 Census counts of the number of
youths under age 18 in three race/ethnicity categories: Hispanic, non-Hispanic black, and total. We could
then calculate the number of non-Hispanic nonblacks under age 18 (Total – Hispanic – non-Hispanic
black). We now needed to get counts for 12- to16-year-olds from these 0- to 17-year-old counts. Even
though more complicated methods (e.g. using the latest age distributions from the Current Population
Survey) could have been used, we used the simplest. We assumed a uniform distribution of youths
between the ages of 0-17, which meant a multiplication adjustment of 5/18. We then adjusted the counts
for the segment sampling rate for HUs in each segment. For example, if we selected half of the listed
housing units, we would expect to find half of that segment’s eligible 12- to 16-year-olds. One final
adjustment is needed, though. We need a screening adjustment because we are not going to find youths
in households where a screener is not completed. Therefore, we need to multiply the counts of youths by
the screening rate. During the sampling process, we assumed a 91 percent screener completion rate
(based on previous surveys), but for this appendix, we used the actual screener completion rate of 94
percent. Here is a summary of how we obtained the expected segment counts (counts below are 1990
Census counts of youths under age 18):
•
•
•
Hispanic 12-16: (Hispanic 0-17) Η .94 Η (5/18) Η (segment sampling rate for HUs)
Non-Hispanic black 12-16: (Non-Hispanic black 0-17) Η .94 Η (5/18) Η (segment sampling rate for
HUs)
Non-Hispanic nonblack 12-16: (Total – Hispanic – non-Hispanic black) Η .94 Η (5/18) Η (segment
sampling rate for HUs)
These expected counts are compared to the actual counts of 12- to 16-year-old youth eligibles found
during screening.
25
Final updating of the NLSY97, Round 1 disposition codes was done in November 1999, in light of all evidence
on eligibility collected to that point in time, including certain evidence collected during Round 2 operations.
Because the tabulations cited in this appendix reflect an earlier generation of the Round 1 disposition codes, there
may be small discrepancies between counts cited here and the final counts cited in the main body of this NLSY97
Technical Sampling Report.
227
Comparison of Two Samples (CX vs. SU), Focusing on Two Strata for SU Sample
As described in Chapter 3, during the selection of the housing units for the SU sample, selected
segments were divided into two strata: “high in Hispanic and/or black youths,” or “low in Hispanic
youths, and low in black youths.” These two strata shall be referred to as HIGH and LOW-LOW for the
remainder of this appendix. Some of the sampling strategy used for NLSY97 was based upon the
expected segment counts of eligible youths described above. For example, a segment was classified as
HIGH or LOW-LOW based upon these (and the housing unit) counts. One concern is how out-of-date
the 1990 Census counts are by the time of the 1997 screening operation for NLSY97. It is certain that the
correlation between 1990 Census counts and actual 1997 counts would be less than 1. This would
indicate that in 1997, the HIGH segments would not be quite as high in minority youths as we would
expect, and the LOW-LOW segments would not be as low in minority youths as we would expect (the
regression effect). In other words, there would be movement of minority youths from HIGH segments to
LOW-LOW segments. One hypothesis that this analysis tries to examine is whether this regression effect
can explain any part of the shortfalls. If the HIGH and LOW-LOW segments were sampled at the same
rate, finding lower counts of minority youths than expected in the HIGH segments, and higher counts
than expected in the LOW-LOW segments might balance out. However, since the HIGH segments were
sampled at ten times the rate of the LOW-LOW segments, the expected shortfalls in minority youths in
the HIGH segments should be about ten times as large as the expected “longfalls” in minority youths in
the LOW-LOW segments.
So, the below analysis compares the actual and expected counts of minority youths in the HIGH
and LOW-LOW segments. With very little additional effort, these counts, as well as actual and expected
counts of non-minority youths, are also analyzed for the CX sample.
Table R-1 below shows the (estimated) expected counts of youths aged 12-16. It should be noted
that the counts of non-Hispanic, nonblack eligibles expected in the SU sample is zero (0) because nonHispanic, nonblack youths are ineligible for NLSY97 in the SU sample. However, the expected number
of non-Hispanic, non-black 12- to 16-year-olds is shown in Table R-1 in order to show how different the
race/ethnicity mix is between the two strata of SU segments. Table R-2 below shows the actual counts of
eligible youths aged 12-16 that were screened.
228
Table R-1 Expected Counts of Youths Aged 12-16 to be Screened
Sample
CX
HIGH
LOW-LOW
TOTAL
SU
TOTAL
Hispanic
Non-Hispanic,
black
Non-Hispanic,
nonblack
Total
1,164
1,431
60
1,491
2,655
1,545
2,000
49
2,049
2,654
7,704
1,153
859
2,012
9,716
10,413
4,584
968
5,552
15,965
Table R-2 Actual Counts of Youths Aged 12-16 Screened
Sample
CX
HIGH
LOW-LOW
TOTAL
SU
Hispanic
Non-Hispanic,
black
Non-Hispanic,
nonblack*
Total
Eligibles
1,026
1,055
47
1,102
2,128
1,175
1,330
41
1,371
2,546
5,134
0
0
0
5,134
7,335
2,385
88
2,478
9,808
TOTAL
*Non-Hispanic nonblacks were ineligible in the SU sample, and were not counted during the screening
process. Therefore, these numbers are shown as zero in Table R-2.
Comparing the numbers in the above tables can be made easier by looking at the percentage of
expected youths that were actually screened. These percentages are shown in Table R-3, which is made
trickier by the SU non-Hispanic nonblacks in Table R-1. Since non-Hispanic nonblacks are ineligible in
the SU, they are removed from the denominator in Table R-3.
Table R-3 Youths Aged 12-16 Actually Screened as a Percent of Expected Youths
Sample
CX
SU
HIGH
LOW-LOW
TOTAL
TOTAL
Hispanic
Non-Hispanic,
black
Non-Hispanic,
nonblack
Total
Eligibles
88.00%
73.64%
78.92%
73.85%
80.05%
76.11%
66.81%
84.32%
67.22%
71.06%
66.71%
N/A
N/A
N/A
66.71%
70.5%
69.7%
81.4%
70.0%
70.4%
Table R-3 shows that both the CX and SU samples screened about 70 percent as many youths as
were expected. However, in the CX sample, the shortfall was concentrated among non-Hispanic
nonblacks, and, to a lesser extent, among non-Hispanic blacks. In the SU sample, of course, non-Hispanic
nonblacks were ineligible. Overall, there was less of a shortfall in the SU sample among Hispanic youths
(73.85 percent vs. 67.22 percent for non-Hispanic blacks).
Table R-3 also allows us to compare the two sampling strata in the SU sample. Recall that
229
because of the time between the 1990 Census and NLSY97, we hypothesized that we would screen fewer
minority youths than we expect in the HIGH segments, but more minority youths than we expect in the
LOW-LOW segments. In fact, there were shortfalls in both sampling strata, but the shortfall in the LOWLOW segments (81.35 percent) was much smaller than in the HIGH segments (69.66 percent).
We now make some complicated calculations to estimate the shortfall (number of youths) caused
by using the seven year-old Census data to draw the NLSY97 sample. We start by calculating the
expected and observed percentages of minority youths in segments that would be classified as HIGH.
Table R-1 shows that we expected to screen 3,431 (1,431 + 2,000) minority youths in the HIGH
segments. If we had sampled LOW-LOW segments at the same rate (we sampled them at one-tenth the
rate), we would have expected 1,090 ([60 + 49] * 10) minority youths in the LOW-LOW segments. This
gives an estimate of 3,431/4,521 = 75.9 percent of the 1990 Census minority youths being in HIGH
segments. A similar calculation using numbers from Table R-2 gives an estimate of (1,336 +
1,054)/(1,336 + 1,054 + 10 * [41 + 47]) = 73.1 percent of 1997 minority youths being in segments that
would be classified as HIGH using 1990 Census data. So, if there had been no other shortfall, we would
have still expected to find only (73.09/75.89) * 3,431 = 3,304 (127 fewer) minority youths in HIGH
segments, but would have expected to find (1-.7309/1-.7589) * 109 = 122 (13 more) minority youths in
LOW-LOW segments. Overall, this is an expected shortfall of 114 minority youths in the SU sample.
The total shortfall in the SU sample was (1,491 + 2,049 – 1,377 – 1,101 = ) 1,062. Therefore, the simple
fact that sampling for NLSY97 was done using seven year-old information explains an estimated
114/1,062 = 10.7 percent of the SU shortfall. Of course, it should be noted that this does not explain why
there were fewer 12- to 23-year-olds screened than youths of the age groups outside this range, which is
the most troubling aspect regarding the shortfalls.
The analyses in this section could be done using a weight for each segment count of expected and
actually screened youths. However, the results would change very little. This is because the proper
weights would be the housing unit base weights for each segment, which are almost uniform within each
of the CX, SU HIGH, and SU LOW-LOW cells of segments.
PSU-Level Analysis
We now look at PSU-level shortfalls, trying to discover any patterns that might be present. We
look at the CX and SU PSUs separately because the natures of the shortfalls are different. In the SU
sample, non-Hispanic, nonblack 12- to 16-year-olds are ineligible for NLSY97. Therefore, any shortfalls
in the SU sample are necessarily among Hispanics and non-Hispanic blacks. In the CX sample, the
shortfalls are most pronounced among non-Hispanic nonblacks. As we did in the HIGH versus LOWLOW analysis, we examine the number of youths aged 12-16 actually screened as a percent of expected
youths. Table R-4 below shows some percentile statistics of these PSU-level percentage variables.
230
Table R-4 shows that fewer youths were screened than were expected in just over 75 percent of
the CX PSUs (Percentile 75 = 99.23 percent), as compared to just under 75 percent of the SU PSUs
(Percentile 75 = 103.76 percent). Means are not provided because they are heavily increased by the
largest (extreme) percentages. A better picture of a “typical” PSU is given by the medians. However,
one odd result is that the SU median percentages for black and Hispanic youths are both much lower than
the SU median percentage for all youths. This (along with the maximum percentages) indicates that these
percentages are highly variable for blacks and Hispanics. However, as shown by the histograms in
Figures R-1 through R-3, the non-minority and all youth percentages have a roughly bell-shaped curve.
Histograms are not shown for the percentages for black and Hispanic youths because the variability is too
large.
The main reason for showing the three histograms in Figures R-1 through R-3 is to indicate that
the percentages do follow a regular pattern when there are enough expected youth cases to form a stable
ratio of actual youths screened over the number of youths expected to be screened. In particular, none of
the three histograms shows any outliers that might indicate a problem, except possibly the zero in Graph
R-3. This is PSU 703, in which the number of expected minority youths was less than 0.5. Three
particular PSUs that might be of interest in Graph R-3 are the three PSUs with a particularly low screener
response rate. These three PSUs, with their percentages of expected youths screened, are PSU 764 (56.43
percent), PSU 795 (65.26 percent), and PSU 862 (67.51 percent). None of these three PSUs is among the
ten lowest (see below) percentages of expected youths screened in the SU sample.
231
Table R-4. Some PSU-Level Percentile Statistics on Percentage Variables (Percentage of Expected Youths Screened)
Median
87.64%
125.72%
81.49%
87.43%
70.79%
75.32%
n/a
87.54%
Percentile
75
117.25%
242.86%
93.57%
99.23%
95.49%
117.61%
n/a
103.76%
5
10
15
Figure R-1 Histogram of CX Non-Minority Youth Percentages
Number of CX PSUs
SU
Pct. of Expected Blacks Screened
Pct. of Expected Hispanics Screened
Pct. of Expected Non-Minorities Screened
Pct. of Expected Youths Screened
Pct. of Expected Blacks Screened
Pct. of Expected Hispanics Screened
Pct. of Expected Non-Minorities Screened
Pct. of Expected Youths Screened
Percentile
25
24.52%
74.89%
68.79%
76.37%
44.77%
.00%
n/a
64.44%
0
CX
Percentile
05
.00%
.00%
42.24%
55.88%
.00%
.00%
n/a
34.83%
Minimum
.00%
.00%
12.29%
44.44%
.00%
.00%
n/a
.00%
0
50
100
150
Non-Minority Youths Age 12-16 Actually Screened as a Percent of Expected Youths
232
Percentile
95
288.56%
867.88%
121.94%
120.90%
157.19%
291.09%
n/a
131.50%
Maximum
1338.76%
1413.49%
140.29%
154.07%
281.93%
2920.64%
n/a
160.05%
10
5
0
Number of CX PSUs
15
Figure R-2 Histogram of CX All Youth Percentages
0
50
100
150
Youths Age 12-16 Actually Screened as a Percent of Expected Youths
6
4
2
0
Number of SU PSUs
8
10
Figure R-3 Histogram of SU All Youth Percentages
0
50
100
Youths Age 12-16 Actually Screened as a Percent of Expected Youths
233
150
We now categorize each PSU in terms of whether the number of youths actually screened
was as many as were expected. Table R-5 divides up the 200 PSUs into the following four
categories:
1)
2)
3)
4)
Zero Expected – The expected count of youths aged 12-16 to be screened was less than 0.5,
Zero Actual – The expected count was greater than 0.5, but none were screened,
Under 100% - The actual count of youths screened was less than the count expected, and
At least 100% - The actual count of youths screened was at least as many as were expected.
Table R-5 divides up the CX and SU PSUs separately, and for all four different percentage
variables (counts of Hispanic, non-Hispanic black, non-Hispanic nonblack, and total youths aged
12-16 actually screened as a percentage of expected youths).
Table R-5 Classification of Four PSU-Level Percentage Variables
CX
SU
Hispanic
Non-Hispanic black
Non-Hispanic nonblack
All Youths 12-16
Hispanic
Non-Hispanic black
Non-Hispanic nonblack
All Youths 12-16
Zero Expected
11
13
0
0
25
8
n/a
1
Zero Actual
5
11
0
0
6
9
n/a
0
Under 100%
22
36
87
78
35
62
n/a
68
Over 100%
62
40
13
22
34
21
n/a
31
In the CX sample, there are eleven PSUs that expected no Hispanic 12- to 16-year-olds
and thirteen different PSUs that expected no non-Hispanic, black 12- to 16-year-olds. However,
there were no CX PSUs that expected zero non-Hispanic, nonblack 12- to 16-year-olds. One of
the PSUs that didn’t expect any Hispanic eligibles actually screened two in (this PSU is still
included in the “Zero Expected” column). There were an additional five CX PSUs that didn’t
screen any Hispanic eligibles, but only one Hispanic eligible was expected in each of these PSUs.
Of the eleven CX PSUs that expected at least one non-Hispanic black, but didn’t screen any in,
one PSU expected around three, three expected around two, and the other seven expected only
one.
Focusing now on the last two columns for the CX, 87 of the PSUs found fewer nonHispanic, nonblack eligibles than were expected. Meanwhile, more than half of the PSUs (62)
found more eligible Hispanic youths than were expected. This seems surprising given that Table
R-3 shows that over the entire CX sample, the number of Hispanic youths screened was only
91.33 percent as many as were expected. Also, discounting the zero expected and zero actual
cells, there are more PSUs (40) that screened more non-Hispanic, black eligibles than were
expected than PSUs (36) that screened fewer. It seems clear that the shortfalls that lead to only
22 CX PSUs screening as many 12- to 16-year-olds as were expected (78 falling short) were
driven by the shortfalls in non-Hispanic, nonblack youths.
In the SU sample, there are twenty-five PSUs that expected no Hispanic 12- to 16-yearolds and eight PSUs that expected no non-Hispanic, black 12- to 16-year-olds. In fact, PSU 703,
chosen to represent counties with very few minority youths, expected zero minority 12- to 16year-olds. One of the PSUs that didn’t expect any Hispanic eligibles actually screened in one.
234
There were an additional six SU PSUs that didn’t screen any Hispanic eligibles, but only one
Hispanic eligible was expected in five of these PSUs; PSU 823 actually expected seven. Of the
nine SU PSUs that expected at least one non-Hispanic black, but didn’t screen any in, two PSUs
expected around four, two expected around three, one expected around two, and the other four
expected only one.
We now examine the PSUs with the largest shortfalls in order to see if there are any
patterns shared by these PSUs. We examine the 10 PSUs for each of the CX and SU samples
with the lowest counts of youths (of all race/ethnicities) aged 12-16 actually screened as a percent
of expected youths. Table R-6 shows the expected counts of youths aged 12-16 to be screened.
Table R-6 Expected Youths Aged 12-16 to be Screened for 10 “Lowest” CX and SU PSUs
CX
SU
PSU 857
PSU 717
PSU 884
PSU 807
PSU 860
PSU 868
PSU 729
PSU 806
PSU 899
PSU 804
PSU 866
PSU 896
PSU 870
PSU 813
PSU 702
PSU 882
PSU 900
PSU 790
PSU 786
PSU 863
Hispanic
6
23
0
7
4
36
0
0
1
1
1
3
0
13
3
0
28
0
23
26
Non-Hispanic,
black
10
36
30
11
8
15
11
2
29
0
16
0
9
24
7
18
4
31
21
0
Non-Hispanic,
non-black
78
75
64
38
42
16
44
64
42
55
26
23
35
7
60
5
23
12
21
51
Total Eligibles
95
134
95
55
54
68
55
66
73
56
43
26
44
45
71
23
55
44
65
77
We have studied the locations of these twenty PSUs and have concluded that they do not
follow a strong geographic pattern. Eight of the ten lowest CX PSUs are urban, two of which are
certainty PSUs. Eight of the ten lowest SU PSUs are also urban, three of which are certainty
PSUs. Louisiana does have one rural parish in each sample among the ten lowest, and three of the
ten lowest CX PSUs are in Tennessee. Table R-7 shows the actual counts of youths aged 12-16
screened.
235
Table R-7 Actual Youths Aged 12-16 Screened for 10 “Lowest” CX and SU PSUs
CX
SU
PSU 857
PSU 717
PSU 884
PSU 807
PSU 860
PSU 868
PSU 729
PSU 806
PSU 899
PSU 804
PSU 866
PSU 896
PSU 870
PSU 813
PSU 702
PSU 882
PSU 900
PSU 790
PSU 786
PSU 863
Hispanic
2
23
1
4
5
27
1
0
0
1
1
1
0
2
2
0
14
0
9
12
Non-Hispanic,
black
2
15
21
12
4
9
2
2
22
0
2
0
3
11
2
7
0
14
11
0
Non-Hispanic,
nonblack
38
28
27
14
21
2
30
38
22
34
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
Total
Eligibles
42
66
49
30
30
38
33
40
44
35
3
1
3
13
4
7
14
14
20
12
It should be noted that of the zeroes shown in Table R-7, only one is a significant part of
the overall PSU shortfall. The expected number of non-Hispanic blacks in PSU 900 was four.
However, even if four non-Hispanic blacks had been screened, they would not have missed this
list of ten lowest by much. For all of the other zeroes shown, the corresponding expected count
was zero (i.e., less than 0.5) or one. Finally, Table R-8 shows the counts of youths aged 12-16
actually screened as a percentage of expected youths.
236
Table R-8 Actual Youths Aged 12-16 Screened as a Percentage
of Expected for 10 “Lowest” CX and SU PSUs
CX
SU
PSU 857
PSU 717
PSU 884
PSU 807
PSU 860
PSU 868
PSU 729
PSU 806
PSU 899
PSU 804
PSU 866
PSU 896
PSU 870
PSU 813
PSU 702
PSU 882
PSU 900
PSU 790
PSU 786
PSU 863
Hispanic
34.15%
100.42%
733.27%
60.95%
119.41%
74.13%
287.49%
n/a
0.00%
144.34%
91.44%
30.55%
0.00%
14.89%
62.95%
0.00%
49.60%
0.00%
38.35%
45.84%
Non-Hispanic,
black
19.26%
41.92%
69.70%
112.42%
52.80%
60.25%
18.59%
90.04%
74.71%
0.00%
12.72%
0.00%
32.13%
45.71%
27.10%
39.30%
0.00%
44.46%
53.65%
n/a
Non-Hispanic,
non-black
48.55%
37.36%
41.90%
37.13%
50.07%
12.29%
68.58%
59.48%
52.76%
61.34%
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
Total
Eligibles
44.4%
49.4%
51.7%
54.6%
55.9%
56.2%
60.2%
60.5%
60.6%
62.0%
17.8%
29.8%
30.9%
34.7%
37.9%
38.6%
43.1%
44.1%
45.5%
45.8%
It is clear that the percentages given in Table R-8 are lower among the SU sample, but this simply
reflects the larger variability in the SU sample caused by smaller numbers of eligibles expected.
Among the CX PSUs, most of the small overall percentages are driven by a low percentage
among non-Hispanic nonblacks. However, PSU 717 also had a large shortfall among nonHispanic blacks, and PSU 868 only expected 16 non-Hispanic nonblacks (compared to 36
Hispanics and 15 non-Hispanic blacks). Also, PSU 857 had only four minority youths screened
(sixteen were expected). Among the SU PSUs, there doesn’t seem to be any pattern in the
shortfalls. Five of the lowest ten PSUs were low in non-Hispanic blacks, three were low in
Hispanics, and two (PSUs 813 and 786) were low in both.
To conclude our PSU-level analysis, we try to look for relationships between the
shortfalls and the screener response rate and two PSU-level socio-economic variables (also
studied in Appendix F). Figures R-4 (CX) and R-5 (SU) below look at the relationship between
the PSU counts of youths screened as a percentage of youths expected and the PSU screener
response rate. These figures show a positive relationship in the CX sample, but no relationship in
the SU. In the CX sample, the shortfalls become greater as the PSU screener response rate
decreases. This is not very surprising because the expected numbers were calculated under the
assumption that the screener response rates would be equal (94 percent) for all segments (and
thus, PSUs). Unlike the analyses with socio-economic variables, we do not perform a simple
regression analysis for screener response rate because we believe that any effect that is shown
here is an artifact of the methodology used to calculate the expected counts of youths to be
screened. It should be mentioned that Figure R-5 does exclude five PSUs with a screener
237
response rate of less than 80 percent. These excluded points fit the pattern shown, though (no
relationship with the percentage of expected youths screened).
Figure R-4. Relationship of PSU Shortfalls to PSU Screener Response Rates – CX
160%
150%
140%
Pct. of Expected Youths Screened
130%
120%
110%
100%
90%
80%
70%
60%
50%
40%
80%
85%
90%
95%
100%
PSU Screener Response Rate
Figure R-5. Relationship of PSU Shortfalls to PSU Screener Response Rates – SU
200%
Pc
t.
of
Ex
pe
cte
d
Yo
uth
s
Sc
re
en
ed
150%
100%
50%
0%
80.0%
85.0%
90.0%
95.0%
100.0%
PSU Screener Response Rate
We next examine the possibility of a relationship between the percentage of expected
238
eligibles actually screened in and socio-economic variables. Figures R-6 (CX) and R-7 (SU)
below look at the relationship between the PSU counts of youths screened as a percentage of
youths expected and the PSU percentage of adults over 25 with a bachelor’s degree, as taken
from the 1990 Census. The two graphs show opposite trends! In the CX sample, PSUs with
more college graduates have greater shortfalls, while in the SU sample, PSUs with fewer college
graduates have greater shortfalls. The two regression lines are:
YCX = 103.6% - .83 * XCX
YSU = 75.6% + .48 * XSU
where Y = Actual counts of youths age 12-16 screened as a percentage of youths expected
and
X= Percentage of adults at least 25 years old with a bachelor’s degree.
The slope is significant for the CX (p=.002), but insignificant for the SU (p=.202).
Comparing the predictions these lines make for a PSU with a low percentage of bachelor’s
degrees (say, 10 percent) with the prediction for a PSU with a high percentage of bachelor’s
degrees (say, 30 percent) gives an idea of the strength of these relationships. With a percentage
college graduates of 30, the CX line predicts that 95.3 percent of the eligibles expected would be
screened, while the SU line predicts that 80.4 percent of the eligibles expected would be screened.
With a 30% percentage of college graduates, the CX line predicts that 78.7 percent of the
eligibles expected would be screened, while the SU line predicts that 90.0 percent of the eligibles
expected would be screened. There seems to be some evidence here that in the CX, PSUs with
more college graduates have larger shortfalls. However, in the SU, PSUs with more college
graduates have smaller shortfalls, even if this trend is not significant.
The third and final relationship we examine is between the percentage of expected
eligibles actually screened in and income. Figures R-8 (CX) and R-9 (SU) below look at the
relationship between the PSU counts of youths screened as a percentage of youths expected and
the PSU median family income, also taken from the 1990 Census. These figures show a positive
relationship in the CX, but no relationship in the SU. In the CX sample, the shortfalls become
greater as the PSU median family income increases. A simple regression line is shown in each of
the figures. For the CX sample, the regression line is:
YCX = 115.9% - .82 * XCX
where YCX = Actual counts of youths aged 12-16 screened as a percentage of youths expected,
and
XCX= Median family income, in thousands of dollars ($1000s).
239
Figure R-6 Relationship of PSU Shortfalls to PSU Education – CX
160%
150%
140%
Pct. of Expected Youths Screened
130%
120%
110%
100%
90%
80%
70%
60%
50%
40%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Percent Bachelors Degree or Higher
Figure R-7 Relationship of PSU Shortfalls to PSU Education – SU
200%
Pct. of Expected Youths Screened
150%
100%
50%
0%
0%
10%
20%
30%
Percent Bachelors Degree or Higher
240
40%
50%
60%
Figure R-8 Relationship of PSU Shortfalls to PSU Median Family Income – CX
160%
150%
140%
Pct. of Expected Youths Screened
130%
120%
110%
100%
90%
80%
70%
60%
50%
40%
$10000
$20000
$30000
$40000
$50000
$60000
Median FamiIy Income, 1990 Census
Figure R-9 Relationship of PSU Shortfalls to PSU Median Family Income – SU
200%
Pct. of Expected Youths Screened
150%
100%
50%
0%
$10000
$20000
$30000
$40000
Median FamiIy Income, 1990 Census
241
$50000
$60000
$70000
Comparing the predictions this line makes for a PSU with a low median family income
(say, $25,000) with the prediction for a PSU with a high median family income (say, $45,000)
again gives an idea of the strength of this relationship. With a median family income of $25,000,
this line predicts that 95.4 percent of the eligibles expected would be screened. With a median
family income of $45,000, this line predicts that 79.0 percent of the eligibles expected would be
screened. We do not show the regression equation for the SU sample because the regression line
is almost completely flat, and the slope is not significantly different from zero.
Conclusions
In the first analysis, comparing the HIGH and LOW-LOW segments, we showed that the
shortfalls in the CX were deepest among non-Hispanic nonblacks, and that in the CX, more nonHispanic blacks were actually screened in than were expected. In the SU sample, the shortfalls
were deeper among Hispanic youths than among non-Hispanic, black youths. Also, the shortfall
in LOW-LOW segments was much smaller than in HIGH segments, as we would expect because
of the seven-year difference between the 1990 Census data used for sampling and the 1997
screening operation. In fact, we estimated that around 12 percent of the SU shortfall can be
explained by this seven-year gap.
In the second analysis, at the PSU level, we showed through histograms that the actual
counts of youths screened in as a percentage of expected youths followed a roughly normal
distribution when the expected counts were large enough to form a stable ratio. In particular, the
percentages for minorities were not stable, but the overall and non-minority percentages were
stable. These histograms also showed a lack of outliers, or problem PSUs.
An examination of the 10 CX and SU PSUs with lowest percentage of expected youths
screened did not show any geographical pattern. These PSUs re-iterated the point that CX
shortfalls were largely among non-Hispanic nonblack youths, but didn’t show any patterns for the
SU sample. We also showed that the SU shortfalls, at the PSU level, were unrelated to the
screener response rate and median family income. There was a trend for PSUs with fewer college
graduates to have higher shortfalls, but the slope of the regression line was not significantly
different from zero. In the CX sample, higher PSU shortfalls were significantly related to lower
PSU screener response rates, higher rates of college graduates (opposite to the trend in the SU
sample), and higher median family incomes. Therefore, there is some evidence that CX (but not
SU) shortfalls might be higher among more affluent PSUs.
In the overall picture, however, the most worrying aspect of NLSY97/PAY97 is the fact
that many fewer 12- to 23-year-olds were in the screening sample than persons under the age of
12 and persons over the age of 23 (the “age hole”). The analyses in this appendix are directed at
looking for patterns in the actual shortfalls of eligible youths, rather than giving any insight into
the “age hole” itself. Finally, all of the analyses in this appendix depend upon the expected
counts of eligible youths; in particular, the assumptions made in calculating them.
242
APPENDIX S
Adjusted Screener Weights
Used for Age Distribution Analysis
243
In this appendix, we describe the adjusted screener weights that were used in Section 5.3
to compare the distribution of all persons 35 years old or younger from NLSY97 screeners to the
March, 1997 Current Population Survey (CPS).
The motivation for these adjusted screener
weights (referred to below as age-distribution weights) is that there are two types of screeners for
which we do not know the age distribution but we do know whether or not there are any ageeligible NLSY97 youths present:
a) proxy/gatekeeper cases – we know that there are no age-eligible NLSY97 youths
present, and
b) “lost” screeners – there are 32 screeners for which we know that there are ageeligible NLSY97 youths present, but the screener data (and therefore the age
distribution) is unavailable.
These screeners have positive screener weights but are missing the age distribution. We treat
these screeners as missing data, and add a separate step to this weighting procedure: adjustment
for age-distribution “nonresponse.”
The age-distribution weights generally follow the first four steps that are described for
youth weights in Section 4. We now proceed through these steps, describing the modifications
made:
Step 1. Base weight with no truncation. The base weight ( W1 ) for the k-th housing unit
in the screening sample is the inverse of the probability of selecting the unit:
W1k =
1
πk
,
where π k denotes the relevant inclusion probability. In Chapter 4, we described how we
truncated the base weights in the supplemental (SU) sample, as well as the missed-housing unit
adjustment. For the purposes of the age-distribution weights, no truncation was performed.
Step 2a. Adjustment for screener nonresponse. The next step is to adjust the screener
base weights for nonresponse to the screening interviews. The nonresponse-adjusted weight
( W2 ) is the base weight ( W1 ) inflated by the inverse of the weighted response rate within an
adjustment cell:
∑W
1j
W2 k =
j ∈Eα
∑αW
W1k
1j
j ∈S
Where Eα is the set of eligible units within the α-th cell and Sα is the set of screener respondents
in the α-th cell. For the youth weights, we adjusted weights within segments. We used the same
approach for these age-distribution weights. This step distributes the W1 weight of the screener
noninterviews across the screener completes within the same segment (or PSU if the segment
244
contains fewer than 25 screener completes). There were no deviations in this step from the
methodology described in Chapter 4.
Step 2b. Adjustment for special disposition codes 71, 74, and 77. The purpose of
these weights is to analyze the age distributions from the household screener data. However, after
step 2a, some households have positive weights but no age distributions. We do have
information, however, on whether or not these households have any youths aged 12-23.
Households with disposition codes 71 and 74 (gatekeepers and proxies) are known to have
reported no youths aged 12-23, but the household may have younger or older persons.
Households with disposition code 77 (screener complete, but data unavailable) are known to have
reported youths aged 12-23, but the report is missing the number of younger or older persons.
Therefore, we can split all screened households into four groups, based on whether we have an
age distribution and whether there are any reported youths aged 12-23. Table S-1 will be helpful
in describing these four groups (the number of cases in each group are in parentheses).
Table S-1 The Four Groups Used in Step 2b of the Age-Distribution Weighting
Age distribution
No age distribution
Screener Completes,
No Youths Reported
Disposition codes 6X
(53,101)
Disposition code 71 and 74 (9,231)
Screener Completes,
Some Youths Reported
Disposition codes 6X (13,046)
Disposition code 77
(32)
The cases with age distributions all have a disposition code in the sixties (6X). Some of these
cases have youths aged 12-23, and some do not. Households with a disposition code of 71 or 74
are more like those with an age distribution but no youths aged 12-23, while households with a
disposition code of 77 are more like those with an age distribution that do have at least one youth
aged 12-23.
Step 2b distributes the W2 weight of the screened households without an age distribution
across the households with age distributions within the same cell, where the cell is determined by
whether or not youths have been reported. In this sense, step 2b is an extra nonresponse weight
adjustment where completes and incompletes are determined by whether an age distribution is
available, and the only two cells given different adjustments are determined by whether any
youths have been reported by that household. We could have used different cells for differing
numbers of youths reported, but it is undesirable to split the 32 cases (households) with the
disposition code of 77 any further.
The adjustment factor for Step 2b is 1.155 in the “no youths reported” cell, and 1.006 in
the “some youths reported” cell. This reflects the fact that very few of the households with
reported youths do not have an age distribution, while the proxies and gatekeepers make up a
sizable proportion of the households with no youths reported.
245
Step 3. Adjustment for subsampling ETP youths. This step is not applicable.
W3k = W2 k .
Step 4. Combine the CX and SU samples using precision weights. At this point, the
sum of the weights in each sample is approximately the total number of U.S. housing units.
However, we would like the sum of the weights for both samples combined to approximate the
total number of U.S. housing units. Therefore, as for the youth weights, we need to use precision
weighting to combine the CX and SU samples.
In creating the youth (NLSY, ETP, and STP) weights, the CX and SU samples were
combined using youth demographics (race/ethnicity, sex, and either age or grade). However, we
desire to combine households, so we can only use household characteristics. The only data we
have for all households is geographic. Therefore, the most logical cells for precision weighting
would be Census region or division. We chose division as the smallest practical cell (state would
be the next smallest). Since the SU sample targets minorities, it consists largely of major cities
and overrepresents the South and West. This will result in greater weight for the SU sample in
the South and West, and relatively smaller weight elsewhere. The λs are shown in Table S-2
below.
Table S-2 λs Used for Precision Weighting for Age-Distribution Weights
Division
New England
Middle Atlantic
East North Central
West North Central
South Atlantic
East South Central
West South Central
Mountain
Pacific
CX (λ)
SU (1- λ)
.93
.92
.93
.91
.89
.94
.77
.84
.83
.07
.08
.07
.09
.11
.06
.23
.16
.17
W4 k = λ$β W3k , i ∈ CX
= (1 − λ$β W3 K , i ∈ SU
as in Chapter 4, where β signifies the Census division domain. W4 k then is the age-distribution
weight.
246
APPENDIX T
Urban/Rural Age Distributions
247
In this appendix, we examine the issue of undercoverage of youths aged 12-23 in the
NLSY97 in more detail by looking at the age distributions of different rural/urban groups. We
classify the households in the screener into different groups based on their rural/urban status, and
compare their age distributions to see if there are significant differences in patterns. We attempt
to identify whether there are specific groups (e.g. rural, urban, suburban) where the problem of
undercoverage of youths aged 12-23 is greater than others.
Classification of NLSY97 households into urban/rural categories: All the households
in the screener, including those obtained by the half-open interval procedure, were matched to the
census block in which they reside. Since PSUs could contain rural, urban and suburban areas, we
have chosen to classify households into rural/urban categories at the block level instead of at a
PSU level. The Bureau of the Census defines urban as “comprising all territory, population, and
housing units located in urbanized areas and in places of 2,500 or more inhabitants outside
urbanized areas.”26 An urbanized area (UA) is a continuously built-up area with a population of
50,000 or more. It comprises one or more places called central places and the surrounding
densely settled areas known as the urban fringe. Territory, population, and housing units not
classified as urban are considered to be rural. The Census classifies all the census blocks as either
rural or urban 27. Since the urban category is a very broad one, which encompasses a variety of
housing environments such as urban centers, suburbs, small cities, and towns, we have further
classified urban areas into three categories:
(1) Central Places of Urbanized Areas -- these correspond to urban centers of urbanized
areas.
(2) Urban Fringe-- parts of urbanized areas which are not parts of central places. This
category corresponds closely to the concept of a suburb in terms of population size,
proximity and economic dependence on the urban center.
(3) Urban areas outside urbanized areas--these are small cities and towns with a
minimum population of 2,500.
Blocks were classified into these categories based on place description codes from the 1990
Census.28
Analysis of age distributions. In Chapter 5, we had identified the undercount for ages
12-23 in the NLSY97 age distributions for the entire sample. This was done by comparing the
number of persons in each age category for the weighted NLSY97 and Current Population Survey
(CPS) age distributions. Tables T-1 and T-2 show the unweighted and weighted29 counts of
persons aged 0-35 for the NLSY97 screening sample in each of the following categories: total
(entire sample), central places of urbanized areas, urban fringes, urban areas outside urbanized
areas, all urban, and rural areas. We calculate the percentage of people in each age category out
of all the individuals of ages 0-3530 using the weighted age distributions shown in Table T-2. This
is done for the overall CPS31 as well as for different rural/urban groups in the NLSY97 and
permits us to compare the different groups which are of different sizes. Age distributions
26
See Rural and Urban Classifications, www.census.gov
1990 Census, Block Statistics, CD90-1B files were used for the rural/urban classification.
28
For details of place description codes see the data dictionary for the 1990 Census 1B files for the
variable PLACEDC.
29
Weighting was done using special screener weights for age distributions.
30
The percentage of x year olds in each group = (number of x year olds/number of 0-35 year olds)*100
31
Data on CPS age distributions for these rural/urban/suburban categories were not available.
27
248
constructed using these percentages are shown in Figures T-1 to T-5. In each case the NLSY97
subgroup is compared to the overall CPS.
When we compare the urban and rural age distributions (Figures T1 and T-2) we find in
both cases that there are spikes at ages 11 and 24 and a hole in the age distribution in the 12-23
age range. The urban percentages gradually decline between ages 12 and 23, while the decline in
the case of the rural age distribution is only slight at age 12, and there is a sharper decline from
age 15 to 23. The gap between the NLSY97 rural age distribution and the CPS is slight in the 1216 age range (NLSY97 eligible youths) and largest towards the end of the interval around ages 20
to 23.
Figure T-1 Weighted Percentages of Persons,
NLSY97 Urban Blocks and CPS
5.00
4.00
"CPS total"
3.00
"NLSY97 urban"
2.00
1.00
0.00
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35
Figure T-2 Weighted Percentages of Persons,
NLSY97 Rural Blocks and CPS
5.00
4.00
"CPS total"
3.00
"NLSY97 rural"
2.00
1.00
0.00
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35
When we look closely at the different subgroups within the urban category, we find that
all of them have a hole in the age distribution in the 12-23 age range. However we observe certain
differences in patterns. Urban centers (central places of urbanized areas) have age distributions
249
(Figure T-3) in which percentages in each age category decline sharply around age 12 and then
remain relatively steady without showing a sharp decline in the 20-23 age range. The size of the
gap between the NLSY97 and the CPS is larger for the younger youths and smaller for older
youths. For the urban fringe (Figure T-4) the proportion of youths falls gradually in the 12-23 age
range till it reaches the 22-23 age group, which is its lowest point. The gap between CPS and
NLSY97 remains more or less constant throughout. Urban areas outside urbanized areas (Figure
T-5) have age distributions with spikes at 11 and 13, followed by a small, gradual decline in the
14-23 age range and a sharper decline in the 20-23 age range.
Figure T-3 WeightedPercentages of Persons,
NLSY97 UrbanCentral Place Blocks andCPS
5.00
"CPStotal"
4.00
"NLSY97 urban central place"
3.00
2.00
1.00
0.00
1 4 7 10 13 16 19 22 25 28 31 34
Figure T-4 Weighted Percentages of Persons,
NLSY97 Urban Fringe Blocks and CPS
5.00
"CPS total"
4.00
"NLSY97 urban fringe"
3.00
2.00
1.00
0.00
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35
250
Figure T-5 Weighted Percentages of Persons,
NLSY97 Urban, Outside UA Blocks and CPS
5.00
4.00
3.00
"CPS total"
2.00
"NLSY97 urban outside UA"
1.00
0.00
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35
Summary
A comparison of the NLSY97 age distributions of various groups based on whether they
are in urban centers, suburbs, other urban areas or rural areas reveals both similarities and
differences between groups. All the rural/urban groups have a drop in the number of people in the
12-23 age group, with spikes at ages 11 and 24, which supports the view that people may have
misrepresented the number of youths in the NLSY97-eligible range in the screener. The shape of
the age distributions do show some variation between groups. The urban age distribution shows a
gradual drop in the percentages of people in each age group in the 12-23 age range, while the
rural age distribution shows a sharper drop, particularly for older youths. In urban centers, the
proportion of persons in each age category shows a decline at age 12 and remains quite steady
around that level until age 23. Both suburban areas and urban areas outside urbanized areas show
a gradual decline in percentages between ages 12 and 23, with the latter showing a more
noticeable depression at the 20-23 range.
When we compare the NLSY97 to the CPS age distributions, there appear to be some
differences in the undercount based on urban/rural status. The undercount is larger for the older
youths in the rural subsample, while it is more evenly distributed over the 12-23 interval in the
urban subsample. Within the urban category, the urban centers show a larger gap in the younger
age group, while in suburban areas the gap is uniform across the 12-23 range. The sub-sample
comprising small cities and towns shows evidence of having a larger gap towards the end of the
12-23 range.
We are primarily interested in the shortfall of 12- to 16-year olds, since this is the group
that was eligible for NLSY97. From a comparison of rural and urban age distributions, it appears
that the 12-16 group is undercounted in urban rather than in rural areas. Within the urban areas,
all categories (urban centers, urban fringes and urban areas outside urbanized areas) have a
shortfall in the 12-16 age range, so we cannot attribute the undercount of youths to one particular
type of urban area.
251
We should interpret these results cautiously for two reasons. The first is that the results
from both the CPS and the NLSY97 are subject to sampling error. The second is that we are using
the overall CPS as a benchmark for comparisons with the NLSY97 for various rural/urban
categories. CPS age distributions could vary across rural/urban groups, just as the NLSY97 does,
in which case observed differences between the CPS and NLSY97 could be due to unobserved
differences in the benchmark age distributions rather than differences in the undercount.
Thus, our analysis of rural/urban age distributions does suggest that there are some
differences in the shapes of the distributions and undercounts of NLSY97 youths for different
types of areas. However, there is no compelling evidence which would allow us to attribute the
undercount to one particular type of rural/urban category.
252
Table T-1 Unweighted NLSY97 Age Distributions for Urban/Rural Groups
Age
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
0-35
Unweighted NLSY97 Counts by Urban/Rural Groups
Total (Rural
and Urban)
Central Place of
Urbanized Area
Urban Fringe of
Urbanized Area
Urban area
outside
Urbanized
Area
1855
2236
2367
2409
2416
2557
2586
2599
2555
2351
2459
2519
2152
2169
2250
2258
2293
1908
1838
1736
1667
1676
1460
1432
2290
2429
2300
2260
2403
2229
2799
2243
2515
2506
2515
2922
684
833
875
889
906
968
937
938
925
827
849
862
696
715
765
723
762
639
668
629
633
681
585
594
976
998
920
924
979
879
1002
852
912
881
896
1005
607
755
732
773
776
846
855
864
834
753
798
826
719
692
738
758
749
623
594
550
560
514
488
507
717
756
733
694
777
677
894
680
834
827
807
1012
150
208
236
215
213
223
228
222
229
213
214
249
187
222
171
190
208
162
159
166
154
144
129
107
191
199
208
200
217
225
270
200
226
247
234
257
1441
1796
1843
1877
1895
2037
2020
2024
1988
1793
1861
1937
1602
1629
1674
1671
1719
1424
1421
1345
1347
1339
1202
1208
1884
1953
1861
1818
1973
1781
2166
1732
1972
1955
1937
2274
414
440
524
532
521
520
566
575
567
558
598
582
550
540
576
587
574
484
417
391
320
337
258
224
406
476
439
442
430
448
633
511
543
551
578
648
81159
29807
26319
7273
63399
17760
253
Total
Urban
Total
Rural
Table T-2 Weighted NLSY97 Age Distributions for Urban/Rural Groups
Age
Weighted NLSY97 Counts by Urban/Rural Groups
Total
(Rural
and Urban)
Central
Place of
Urbanized
Area
Urban
Fringe of
Urbanized
Area
Urban
outside
Urbanized
Area
Total
Urban
Total
Rural
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
2577942.4
3030546.7
3303989.6
3372036.5
3279069.3
3484256.5
3498752.5
3537802.1
3409837.6
3168310.3
3288936.9
3404902.3
2669284.1
2697544.3
2740386.6
2726194.2
2735635.1
2372761.3
2211489.2
2149833.7
2079189.4
2164810
1840284.5
1816832.1
3176756.6
3484870.6
3303172.9
3250871.9
3388165
3173183.6
4199563
3279151.6
3557179.3
3611873.1
3563318.6
4205061.4
793196.8
942508.9
997411.3
1038285.7
1015887.8
1124976.8
1043523.4
997838.7
1033530.6
916472.1
942132.2
948501.4
699864.0
715034.1
784878.5
682389.1
752407.0
614459.4
678731.7
663318.8
682696.0
779232.6
648688.6
666475.5
1156853.0
1252777.3
1170724.8
1130640.7
1183015.8
1097494.6
1296088.7
1065376.3
1096069.2
1112707.8
1064149.7
1221751.1
891727.9
1119918.6
1073595.1
1164132.4
1123965.7
1241171.9
1255291.4
1314468.7
1166363.5
1089028.5
1165878.7
1212747.5
943516.5
920322.4
959233.7
993285.0
971111.2
833778.8
765850.7
741793.4
757931.0
724035.9
659058.4
684615.1
1081201.1
1117207.3
1093945.1
1102752.6
1192717.3
1018209.6
1409662.7
1081843.9
1276855.2
1287326.0
1241092.0
1496538.9
228154.0
288344.0
337868.3
302729.5
295707.5
300436.5
310948.2
325988.9
334631.9
278958.2
275166.8
316522.1
230317.7
278227.9
189487.7
216042.8
219633.3
198441.0
182337.4
188720.3
180287.5
172626.6
150084.4
135996.2
269606.8
298394.0
313082.1
289032.2
317321.7
326756.1
432546.1
295297.0
324562.0
337427.2
318473.1
404254.59
1913078.6
2350771.5
2408874.7
2505147.6
2435561.1
2666585.2
2609762.9
2638296.3
2673988.2
2284458.8
2383177.7
2477771.0
1873698.1
1913584.4
1933600.0
1891716.9
1943151.4
1646679.1
1626919.8
1593832.6
1620914.5
1675895.0
1457831.5
1487086.8
2507660.8
2668378.6
2577752.0
2522425.6
2693054.9
2442460.3
3138297.5
2442517.2
2697486.3
2737461.0
2623714.8
3122544.4
664863.7
679774.8
895114.9
866888.9
843508.2
817671.3
888989.5
899505.8
875311.6
883851.6
905759.2
927131.3
795586.0
783959.9
806786.6
834477.3
792483.7
726082.2
584569.4
556001.2
458274.9
488915.0
382453.0
329745.4
669095.7
816492.0
725420.9
728446.3
695110.1
730723.3
1061265.6
836634.5
859692.9
874412.1
939603.8
1082516.9
0-35
109753794
34010089.9
38172173.8
9864411.2
82046675.0
27707119.3
254
| File Type | application/pdf |
| File Title | CHAPTER 1 |
| Author | moore-whitney |
| File Modified | 2013-01-30 |
| File Created | 2003-04-24 |