Parts B-E ELS 2002 3rd-Follow-up 2012
Parts B-E ELS 2002 3rd-Follow-up 2012.docx
Education Longitudinal Study (ELS) 2002 Third Follow-up 2012 (Full Scale)
OMB: 1850-0652
December 2011
Education
Longitudinal Study: 2002
(ELS:2002)
Third Follow-up 2012 Full-scale Study
OMB Supporting Statement
Parts B, C, D, and E
OMB# 1850-0652 v.8
National Center for Education Statistics
Institute of Education Sciences
U.S. Department of Education
TABLE OF CONTENTS
Part B: Statistical Sampling Design
Part C: Survey Item Justification
Part D: Linkages to Extant Data Sources
Part E: Field Test Response Propensity Modeling Experiment and Case Assignment
December 2011
Education
Longitudinal Study: 2002
(ELS:2002)
Third Follow-up 2012 Full-scale Study
OMB Supporting Statement
Part B
OMB# 1850-0652 v. 8
National Center for Education Statistics
Institute of Education Sciences
U.S. Department of Education
TABLE OF CONTENTS
Section Page
B. Collection of Information Employing Statistical Methods B-2
B.1 Respondent Universe B-2
B.2 Statistical Procedure for Collecting Information B-3
B.3 Methods for Maximizing Response Rates B-3
B.4 Reviewing Statisticians and Individuals Responsible for Designing and Conducting the Study B-9
Exhibits
Number Page
Exhibit B-1. Third
Follow-up Full-Scale Data Collection Schedule and
Projected
Response by Survey Mode B-6
Exhibit B-2. RTI Consultants on Statistical Aspects of ELS:2002 B-9
Exhibit B-3. Other Contractor Staff Responsible for Conduct of ELS:2002 B-9
The respondent universe for ELS:2002 is described as is the sampling and statistical methodology proposed for the third follow-up full-scale study. We also describe methods for maximizing response rates, the special tests of procedures and methods, and the statisticians and other persons responsible for designing and conducting the study.
B.1 Respondent Universe
The target populations of the third follow-up full-scale study are the 2002 sophomore cohort and the 2004 senior cohort. The sophomore cohort consists of those students who were enrolled in the 10th grade in the spring of 2002, and the 12th-grade cohort consists of those students who were enrolled in the 12th grade in the spring of 2004. The sophomore cohort includes students who dropped out of school between 10th and 12th grade. Response rates through the second follow-up were high enough to expect large returns in the third follow-up of the study’s primary groups of interest: dropouts from high school, non-college bound high school graduates, and college-bound high school graduates (i.e., those who enrolled in college at time of last contact). The overall weighted response rate was 88 percent (89 percent unweighted). Response rates for respondents who had ever reported a dropout episode (~1200 cases) were about 83 percent for the second follow-up. Non-college bound students also make up a large part of the sample; about 30 percent of second follow-up respondents reported never enrolling in postsecondary education as of 2006.
The study population for the third follow-up consists of sample members who were enrolled in the 10th grade in the base-year study, or were enrolled in the 12th grade in the first follow-up study, and participated, at minimum, in either the base year or first follow-up rounds. The target population comprises all living members of the third follow-up 10th grade and 12th grade cohorts. In addition to the deceased, and to sample members who participated neither in the base year nor first follow-up, groups not followed for efficiency or practicality reasons include E sample members who consistently (across rounds) were physically or mentally incapable of participation. Post-tracing, the 2012 questionnaire will not be fielded to study members who are found to be: out of country, institutionalized, incarcerated, or “newly” incapable (having suffered a major cognitive or physical impairment). Sample members who have asked that their data be withdrawn from the study are eligible members of the sample frame, but will be treated as permanent nonrespondents and not fielded. Such eligible but nonfielded cases will be accounted for in the weighting and count against the response rate.
B.2 Statistical Procedure for Collecting Information
The ELS:2002 third follow-up full-scale sample will consist of approximately 16,200 sample members who were sophomores in 2002 or seniors in 2004 or both. The procedures that will be employed will target an overall response rate of 90 percent.
B.3 Methods for Maximizing Response Rates
Our plan to maximize response rates focuses on two related goals of the ELS:2002 third follow-up data collection: (1) successful locating of sample members, and (2) a successful data collection.
The first goal is successfully locating and contacting sample members by successfully implementing a tracing and sample maintenance plan. A successful locating effort is dependent on a multitude of factors including the characteristics of the population, the age of the locating information for the population, and the completeness and accuracy of that information. The locator database for the cohort includes critical tracing information for most of the sample members, including their previous residences and telephone numbers. Moreover, Social Security numbers are available for 91 percent.
The tracing and sample maintenance plan includes, initially, the use of batch tracing services without direct respondent contact. Subsequently, direct mailings to sample members or their parents will take place. Batch tracing is a relatively low-cost method of updating addresses and telephone numbers for sampled individuals. The two primary batch tracing services include National Change of Address (NCOA) and Phone Append. NCOA will provide updated addresses for sample members, especially those who have recently moved. Phone Append will confirm or update the telephone number matched to each sample member at their most current known address. Cases whose contact information is not confirmed or updated by these steps will be sent for Accurint batch tracing. Using the updated address information obtained from these steps, the batch tracing activity will be followed with a direct mailing to sample members and their parents.
The inclusion of parents is an important support to the tracing and contacting goals of the study. Parents have been deeply involved in the study since its onset: parent permission was sought for student participation in the base year and first follow-up, and a parent survey was conducted in the base year. Parents have also previously been contacted for tracing and locating purposes. The age of the sample population makes it very likely that many of the sample members have recently gone through major life transitions. For students who entered postsecondary education (74 percent at the second follow-up), they are now more than 7 years past their initial entry point into postsecondary education and most students who persisted in their education are likely to have completed their certificate, associate’s degree, or bachelor’s degree programs. Sample members are unlikely to still be attending the undergraduate institution they were attending at the time of the second follow-up. Most of the sample members are now 25 or 26 years of age and have entered the workforce and started their careers. Some have married and changed names, making the task of updating their locating information somewhat more difficult. Another consideration is that 46 percent of adults aged 25 to 29 live in wireless telephone-only households,1 creating a greater locating challenge. For these reasons, we expect sample members’ parents to be a useful source for updating contact information for their children. Because parents’ residences will likely be more stable than their children’s, they will be very important to the tracing effort for a significant number of sample members. For this reason, the batch tracing activities and the sample maintenance mailings described in this plan will include parents. Also, study contacting materials will be produced in English, Spanish, and, on an at-need basis, up to four Asian languages, to facilitate contacts with parents whose first language is not English. Panel maintenance activities to facilitate sample locating were described fully in an earlier submission to OMB that obtained permission to contact sample members or their parents. These panel maintenance activities are currently ongoing.
We will also conduct batch tracing and intensive in-house tracing at RTI during data collection for cases we cannot locate. Immediately before beginning intensive tracing, batch tracing will be conducted on cases determined to have no good phone number in an attempt to keep costs as low as possible. If that step is unsuccessful, the case will be flagged for intensive tracing. The goal of intensive tracing is to obtain a telephone number at which the sample member can be reached. Tracing procedures may include (1) Directory Assistance for telephone listings at various addresses, (2) criss-cross directories to identify (and contact) the neighbors of sample members, (3) calling persons with the same unusual surname in small towns or rural areas to see if they are related to or know the sample member, and (4) contacting the current or last known residential sources such as the neighbors, landlords, and current residents of the last known address. Other more intensive tracing activities could include (1) database checks for sample members, parents, and other contact persons, (2) credit database and insurance database searches, (3) drivers’ license searches through the appropriate state departments of motor vehicles, (4) calls to colleges, military establishments, and correctional facilities to follow up on leads generated from other sources, (5) calls to alumni offices and associations, and (6) calls to state trade and professional associations based on information about field of study in school and other leads. The same vendors that were mentioned in the approved panel maintenance submission to OMB will be utilized.
Communication with sample members prior to and during data collection will promote cooperation by offering multiple methods for them to contact us (see Appendix 2 for contacting materials). The lead letters will provide the study website where sample members can find more information about the study and participating (by completing the questionnaire) via the Web. The letters will also provide telephone numbers respondents can use to contact RTI staff. Follow-up telephone calls will be used to determine whether sample members have in fact received the materials we will have mailed to them or visited the study website. Contact materials and the web site have been designed so as to be attractive and informative in a manner aimed to enhance participation in ELS:2002.
For those with Internet access, the first opportunity will be to complete a self-administered Web questionnaire. Instructions for completing the questionnaire via the website will need to be as clear and simple as possible to facilitate maximum web participation. Because it is not possible to ensure that all potential respondents will fully understand the instructions or that computer glitches will not occur, we will make it clear to potential web respondents that help desk staff will be available to them if and when they need them. Simply offering sample members a self-administered option is likely to increase response by allowing respondents greater discretion over how and when they can participate.
Despite the many advantages of offering a web survey mode, previous studies indicate that response rates are somewhat higher in interviewer-administered surveys than in self-administered surveys. Further methods of contact will include phone interviews and in-person interviews to increase contact with sample members over mail and Web contacts. Given the need to achieve high response rates, the interviewer training will focus considerable attention on enlisting cooperation. A large portion of the interviewer training for the third follow-up will concentrate on the most effective techniques for increasing participation. The two most important techniques on which interviewers will be trained are maintaining interaction with sample members and tailoring their approach to address the specific situation or concerns of potential respondents. An important part of these efforts is not only to highlight the importance of ELS:2002, but also to emphasize the importance of each respondent’s participation in the third follow-up survey. Exhibit B-1 summarizes the third follow-up full-scale data collection schedule and projected responses by survey mode.
Exhibit B-1. Third Follow-up Full-Scale Data Collection Schedule and Projected Response by Survey Mode
Survey Mode |
Web Before CATI Prompting |
Web After CATI Prompting |
CATI |
CAPI |
Dates |
July 2 – July 29 |
July 30 – Jan 14 |
July 30 – Jan 14 |
Sept 10 – Jan 14 |
Respondents |
4,198
Respondents |
2,316
Respondents |
5,212
Respondents |
1,303
Respondents |
Cumulative Respondents |
3,371
Total |
5,006
Total |
12,980
Total |
14,477
Total |
The use of cell phone calling and text messaging is a relatively new means for contacting sample members. Little research has been conducted on the effects of text messaging on participation rates. Research conducted by Brick et al. suggests that text messaging as a method of prenotifying sample members has nearly equal response rates to control group counterparts (Brick, Brick, Dipko, Presser, Tucker, and Yuan 2007). According to Lambries et al., those households using primarily cell phones required more attempts to contact than those using both landline and cell phones and those using landline only (Lambries, Link and Oldendick 2006). Households that used cell phones primarily showed differences of 1.1 more attempts than landline-only household and 0.8 more attempts than both landline and cell phone households.
However, text messaging has some advantages as the first means of contacting sample members. Text messaging may help identify working numbers and, in turn, increase the efficiency of the calling process (Steeh, Buskirk and Callegaro 2007). The research by Steeh et al. concludes that text messages have two advantages as the first means of contact: outcome rates are substantially improved and information about the working status of the number is obtained. Further research in this field is needed to better understand the effects of cell phone calling and text messaging on participation rates. The previously approved panel maintenance mailings include a question asking sample members if they would like to receive a text message when data collection is about to begin.
RTI will draw upon its most effective strategies for increasing survey participation. In light of the field test results and recent discussions between OMB and NCES, ELS:2002 will move toward implementing a survey design with more responsive and adaptive features. ELS:2002 will move beyond response rates and toward a metric that better indicates bias reduction and stability in key estimates. The goal of this new metric will be to produce less-biased and more efficient estimates of key population parameters.
For ELS:2002/12, we propose to use a Mahalanobis distance function calculated at three points in data collection to identify nonrespondent cases which are most unlike existing respondents.
The following variables will be used to calculate the Mahalanobis distance:
Substantive data already collected/available from base year, first and second follow-up: enrollment status, parent’s education, high school completion status, test score quartile, income quartile, native English speaker, school control, school locale, postsecondary enrollment, current work status, and whether the case was in school in grade 12.
Paradata from current and prior rounds: response status from prior rounds, whether sample member contacted the help desk, whether sample member logged in but did not complete the questionnaire, number of contact attempts in the early data collection period, whether sample member made an appointment, and whether sample member told interviewer they would do the web interview. The Mahalanobis function incorporates a measure of the likelihood of ultimate response among current nonrespondents. Therefore, paradata are very important to include with the substantive data for factoring into the case-selection process thereby optimizing the selection itself.
Mahalanobis distance can be defined as the distance between a case and some group centroid. Therefore, cases with larger distance scores can be thought of as cases demonstrating large differences from the group. That is, these cases would be characterized by differences, for example, in their enrollment status, parent’s education, high school completion status, etc.
Identifying these cases and presenting the specifically-targeted nonresponding cases with a higher incentive will in turn attempt to boost their participation and potentially reduce bias in estimates and also improve analytic power through higher sample sizes for these groups of cases of analytic interest. Cases identified for targeting via their Mahalanobis distance (nonrespondent cases with distance scores furthest from the mean respondent) will receive a $55 incentive – rather than the $25 base incentive – in an attempt to encourage their participation. We should note that ever-dropout cases are a critical set of cases and will receive $55 from the start of data collection rather than being included in the Mahalanobis distance measure selection process during data collection. The approach with ever-dropouts is based on the particular difficulty gaining cooperation from this important subgroup as demonstrated during the F3 field test and prior rounds of data collection.
We propose to measure the distance functions at three points during data collection. At these points, the cases with the largest distance scores will be offered the $55 incentive while the $25 base incentive will remain intact for all other cases. Other activities that will be conducted to locate and interview targeted cases include performing pre-data collection intensive tracing, pursuing the cases in person with field locator/interviewers, and sending a $5 prepaid incentive near the end of data collection. The case-identification for more-intensive effort will be done at three points during data collection.
Our first target-case-selection point will be right before outbound CATI is scheduled to begin, 4 weeks into data collection.
The second point will be right before the CAPI period begins, 9 weeks into data collection.
The third point will be just prior to the pre-paid incentive period, approximately 8 weeks prior to the end of data collection.
At each juncture, the cases targeted will be those with the largest distance scores but not targeted in the prior phase(s). Also at the third and final point, cases identified for targeting will receive a $5 prepaid incentive in addition to the $55 incentive and other non-monetary activities, listed above. At this late point in data collection, these cases are likely to be the most critical for reducing the final mean distance function scores between respondents and nonrespondents, thus potentially improving analytic power in terms of sample size and reducing bias in estimates. We believe it prudent to be highly strategic with the use of prepaid incentives, which are normally very effective. Again, cases not targeted will continue to be offered the $25 incentive and will not be targeted for the more-intensive non-monetary activities.
Case targeting will be based on distance scores and anticipated yield. Based on the proportion of interviews projected across each of the phases of data collection, the distance scores will be segmented such that the following approximate numbers of cases are to be targeted at each cut point: 915 cases at the first cut point, 760 cases at the second cut point, and 830 at the third cut point. At each of these time points, we will evaluate the Mahalanobis values among the respondents and nonrespondents to identify the target cases.
In summary, plans include classifying all cases that have ever dropped out of high school as a special group to target prior to data collection and offering them a $55 incentive from the start of data collection. Using the Mahalanobis distance function, special target cases will be identified at the three points during data collection. These cases will be offered an increased incentive of $55 in an effort to encourage participation and reduce the risk of bias.
The key to a successful third follow-up data collection will be combining all survey design elements into a comprehensive and effective strategy.
B.4 Reviewing Statisticians and Individuals Responsible for Designing and Conducting the Study
A number of individuals have consulted with NCES and RTI on the design and analysis plans for the ELS:2002. Members of the TRP have been described in Part A of this submission. In addition, Jeffrey Owings, Associate Commissioner for the Elementary/Secondary and Library Studies Division at NCES, has reviewed and approved the statistical aspects of the study. Other statistical reviewers at NCES include the project officer, Elise Christopher; the NCES Chief Statistician, Marilyn Seastrom; and the then-Disclosure Review Board chair, Neil Russell. Exhibit B-2 provides the names of RTI consultants on statistical aspects of ELS:2002, while Exhibit B-3 lists other principal RTI professional staff assigned to the study.
Exhibit B-2. RTI Consultants on Statistical Aspects of ELS:2002
Name |
Affiliation |
James Chromy |
RTI |
Steven J. Ingels |
RTI |
Daniel J. Pratt |
RTI |
John Riccobono |
RTI |
Peter H. Siegel |
RTI |
David Wilson |
RTI |
Exhibit B-3. Other Contractor Staff Responsible for Conduct of ELS:2002
Name |
Affiliation |
Chris Alexander |
RTI |
Ben Dalton |
RTI |
Donna Jewell |
RTI |
Erich Lauff |
RTI |
Tiffany Mattox |
RTI |
Jim Rogers |
RTI |
December 2011
Education
Longitudinal Study: 2002
(ELS:2002)
Third Follow-up 2012 Full-scale Study
OMB Supporting Statement
Part C
Survey Item Justification
OMB# 1850-0652 v.8
National Center for Education Statistics
Institute of Education Sciences
U.S. Department of Education
Reflecting the outcome of TRP, agency and contractor review and recommendations, a grid of questionnaire changes, and their justifications, appears below.
Item |
Source |
Status |
Old Wording (if revised) |
New Wording |
Justification |
F3ACTCURR |
ELS F3 |
Revised |
Are
you currently... |
Are
you currently... |
To improve clarity, revised parenthetical associated with “working for pay at one or more full-time jobs” and “working for pay at one or more part-time jobs” items. |
F3ACTSAME F3ACTJUNE F3ACTSAMEW F3ACTJUNEW F3ACTSAMES F3ACTJUNES F3ACTSAMEH F3ACTHOMEH |
ELS F3 |
Revised |
F3ACTSAME: Were your work and school activities the same during the last week in June 2011 the same as they are now?
F3ACTJUNE: During
the last week in June 2011, were you... |
F3ACTSAMEW: Were
your work and/or military activities during the last week in June
2012 (if any) the same as they are now?
F3ACTJUNEW: During
the last week in June 2012, were you…
F3ACTSAMES: Were
your school activities during the last week in June 2012 (if any)
the same as they are now?
F3ACTJUNES: During the last week in June 2012, were you… Response options: yes/no for each item below • taking courses at a two- or four-year college, including graduate or professional schools? • taking courses at a vocational, technical, or trade school?
F3ACTSAMEH: Were
your home activities (such as keeping house and/or caring for
dependents) during the last week in June 2012 (if any) the same
as they are now?
F3ACTJUNEH: During
the last week in June 2012, were you… |
For clarity, (1) split F3ACTSAME into F3ACTSAMEW, F3ACTSAMES, and F3ACTSAMEH; and (2) split F3ACTJUNE into F3ACTJUNEW, F3ACTJUNES, and F3ACTJUNEH. |
F3PSLAST |
ELS F3 |
Revised |
Which institution did you last attend? |
Which institution [did you last attend / are you currently attending]? |
Revised wording to more appropriately accommodate those still enrolled. |
F3PSCREDTYPE |
ELS
F3 |
Revised |
What
type of degree or certificate did you receive from [school
name]? |
What
type of degree or certificate did you receive from [school
name]? |
Revised set of response options to better distinguish the "certificate" option; new set of response options is borrowed from NPSAS:12. |
F3PSINCCRED |
ELS F3 NPSAS:12 |
Added |
Not asked. |
What type of degree or certificate were you pursuing when you were last attending [school name]? -or- What type of degree or certificate are you currently pursuing at [school name]? 1=Undergraduate certificate or diploma (usually less than 2 years), including those leading to a license (for example, cosmetology) 2=Associate’s Degree (usually a 2-year degree) 3=Bachelor’s Degree (usually a 4-year degree) 4=Post-baccalaureate certificate 5=Master’s Degree 6=Post-master’s certificate 7=Doctoral Degree – research/scholarship (for example, PhD., EdD., etc.) 8=Doctoral Degree – professional practice (including: chiropractic, dentistry, law, medicine, optometry, pharmacy, podiatry, or veterinary medicine) 9=Doctoral Degree – other |
Added new question on intended credential type, to be administered to (1) those with postsecondary attendance but no credential, or (2) those currently enrolled. |
F3OTHCREDTYP |
ELS
F3 |
Revised |
[You
have already told us about: |
[You
have already told us about: |
Revised set of response options to better distinguish the "certificate" option; new set of response options is borrowed from NPSAS:12. |
F3PSIMPACT |
ELS F3 |
Revised |
[Have
you participated/Did you participate] in any of the following as
a part of your [college/undergraduate] education? |
[Have
you participated/Did you participate] in any of the following as
a part of your [college/undergraduate] education? |
Revised wording for 1st item per updated wording in original source survey (NSSE); also cut the following two items: "Honors program" (it is likely based on the student’s previous achievement or ability and thus is a selection issue, and the qualities of honors programs can vary greatly from one campus to the next), and "Case Studies competitions" (very few 'yes' responses). |
F3PSPREPLIFE |
ELS F3 |
Revised |
How
important would you say your [undergraduate education was
in/college education was in/college attendance has been in]
preparing you for the following aspects of your life? |
How
important would you say your [undergraduate education was
in/college education was in/college attendance has been in]
preparing you for the following aspects of your life? |
Removed last item ("Your overall quality of life") per TRP recommendations. |
F3CREDSAT |
ELS F3 |
Dropped |
How
satisfied are you with the following aspects of your
[undergraduate education/college education/college attendance]? |
Dropped. |
Dropped per TRP recommendations. |
F3WHYNOCRED |
ELS F3 |
Revised |
Earlier
you indicated that you are no longer enrolled in any school and
that you did not obtain a degree or certificate. Which of the
following are reasons you left school? Response options: yes/no for each item below
|
Earlier you indicated that you are no longer enrolled in any school and that you did not obtain a degree or certificate. Which of the following are reasons you left school? -or- Earlier you indicated that you had once attended a 4-year school, but did not obtain a credential from a 4-year school. Which of the following are reasons you left a 4-year school without completing a credential? Response options: yes/no for each item below
|
The ‘or’ clause in the question stem has been revised to account for the fact that some 4-year institutions grant less-than-4-year credentials. Also, during TRP discussions, it was agreed that: (1) “financial reasons” be split into “you couldn’t afford to continue going to school” and “you’d rather work and make money than continue going to school” to obtain further detail/specificity; (2) “academic problems” and “not satisfied with the program, school, campus, or faculty” be dropped along with F3CREDSAT; (3) “difficulty in completing requirements for your program, including developmental or remedial courses” be added, since current research indicates challenges involved in completing developmental courses are a primary reason for dropping out of postsecondary school; (4) “moved from the area”, “decided to take time off from studies”, and “enrollment did not suit your lifestyle, or you were bored with school” be dropped since they are too vague to be useful; and (5) “the school or program closed, or lost accreditation” be dropped due to low frequency of ‘yes’ responses. |
F3EDEXP |
ELS F3 NPSAS:12 |
Revised |
What is the highest level of education you ever expect to complete? • Less than high school graduation • GED or other high school equivalency only • High school diploma only • Completion of a 1- or 2-year program in a community college or vocational school • Bachelor's degree • Master’s degree or equivalent • Ph.D., professional doctorate, or equivalent • Don’t know |
What is the highest level of education you ever expect to complete? 1=Less than high school graduation 2=GED or other high school equivalency only 3=Undergraduate certificate or diploma (usually less than 2 years), including those leading to a license (for example, cosmetology) 4=Associate’s Degree (usually a 2-year degree) 5=Bachelor’s Degree (usually a 4-year degree) 6=Post-baccalaureate certificate 7=Master’s Degree 8=Post-master’s certificate 9=Doctoral Degree – research/scholarship (for example, PhD., EdD., etc.) 10=Doctoral Degree – professional practice (including: chiropractic, dentistry, law, medicine, optometry, pharmacy, podiatry, or veterinary medicine) 11=Doctoral Degree – other 12=Don’t know |
Revised set of response options to match those used elsewhere in the questionnaire (new set of response options is from NSPSAS:12). |
F3LOANOWE |
NPSAS:12 |
Revised |
How much of this amount that you borrowed do you still owe? Please do not include money borrowed from family or friends. |
What is the total amount of money you borrowed in student loans since high school? Do not include parent PLUS loans or any money borrowed from family or friends in your answer. (If you are unsure of the amount, please provide your best guess.) |
Per TRP, "total amount borrowed" is more valuable to collect than is "amount still owed" (see also F3LOANOWE below); Actual wording from NPSAS:12 is “What is the total amount of money you borrowed in student loans for the 2010-2011 school year? Do not include parent PLUS loans or any money borrowed from family or friends in your answer. (If you are unsure of the amount, please provide your best guess.)” |
F3LOANREPAY |
BPS:09 |
Added |
Not asked. |
Are
you currently repaying any of your educational loans? |
Added to serve as a gate question for F3LOANPAY; actual BPS:09 wording is “Are you currently repaying any educational loans for your undergraduate education?” |
F3LOANPAID |
ELS F3 |
Dropped |
Has
any of your student loan debt been... Before
we continue to the next question, we would like to know if you
had any difficulty understanding or answering this question so
that we may improve it for future surveys. Did you have any
difficulty such as not understanding question wording, being
uncertain of the meaning of certain terms or response choices, or
not having the information needed to answer the
question? If yes, please describe any difficulty you had. Please be as specific as possible. |
Dropped. |
Cut per TRP recommendations. |
F3LOANAFFECT |
ELS F3 |
Revised |
Did your student loan debt influence your employment plans and decisions in any of the following ways? Response options: yes/no for each item below • You took a job outside your field of study or training • You took a less desirable job • You had to work more hours than desired • You had to work more than one job at the same time • Student loan debt has influenced your plans or decisions in some other way |
Did your student loan debt influence your employment plans and decisions in any of the following ways? Response options: yes/no for each item below • You took a job outside your field of study or training • You took a less desirable job • You had to work more hours than desired • You had to work more than one job at the same time |
Dropped "other-specify" item (“Student loan debt has influenced your plans or decisions in some other way”) for main study. |
F3GRANT |
ELS F3 |
Revised |
While
attending [school name], did you receive any grants or
scholarships? |
During
your first year at [1st PS school attended], did you receive any
grants or scholarships? |
During TRP discussions, it was agreed that questionnaire time would be better used asking for grants/scholarships received from first (and last) schools/years attended, rather than ask for grants/scholarships received from each school attended. |
F3GRANTAMT |
ELS F3 |
Revised |
Approximately
how much did you receive in grants and/or scholarships while
attending [school name]? Before
we continue to the next question, we would like to know if you
had any difficulty understanding or answering this question so
that we may improve it for future surveys. Did you have any
difficulty such as not understanding question wording, being
uncertain of the meaning of certain terms, or not having the
information needed to answer the question? If yes, please describe any difficulty you had. Please be as specific as possible. |
How
much of the tuition and fees for your first year at [1st PS
school attended] did those grants or scholarships pay
for? |
Cognitive
interviews (as well as the "did you have any difficulty
answering" follow-up question used in FT) suggest asking for
proportions may yield more accurate responses than does asking
for an exact amount. |
F3GRANTL |
ELS F3 |
Revised |
While
attending [school name], did you receive any grants or
scholarships? |
During
your last year at [school name], did you receive any grants or
scholarships? |
During TRP discussions, it was agreed that questionnaire time would be better used asking for grants/scholarships received from first (and last) schools/years attended, rather than ask for grants/scholarships received from each school attended. |
F3GRANTAMTL |
ELS F3 |
Revised |
Approximately how much did you receive in grants and/or scholarships while attending [school name]? |
How
much of the tuition and fees for your last year at [last PS
school attended] did those grants or scholarships pay
for? |
Cognitive interviews suggest asking for proportions may yield more accurate responses than does asking for an exact amount. |
F3LICENSE |
ELS F3 |
Revised |
Now
we would like to ask you about professional certification and
licensure. Do you have a current professional certification, a
professional license, or a state or industry license? Before
we continue to the next question, we would like to know if you
had any difficulty understanding or answering this question so
that we may improve it for future surveys. Did you have any
difficulty such as not understanding question wording, being
uncertain of the meaning of certain terms, or not having the
information needed to answer the question? If yes, please describe any difficulty you had. Please be as specific as possible. |
Now
we would like to ask you about professional certification and
licensure. Do you have a current professional certification, a
professional license, or a state or industry license? |
Removed the "did you have any difficulty answering" follow-up question, which was only intended for eliciting information to potentially improve questionnaire for FS. |
F3LICENSEBY |
ATES |
Added |
Not asked. |
Were you certified or licensed by… Response options: yes/no for each item below • your state? • a professional organization? • an industry, company, or some other organization? |
TRP discussions indicated knowing who granted the license/certification would be a more important question to include than would the licensure/certification questions recommended for cutting (below). |
F3MULTILICENSE |
ELS F3 |
Dropped |
Do you have more than one certification or license? |
Dropped. |
Deemed by the TRP to be relatively less important than the licensure/certification question added above (F3LICENSEBY). |
F3NUMLICENSE |
ELS F3 |
Dropped |
How many do you have? |
Dropped. |
Deemed by the TRP to be relatively less important than the licensure/certification question added above (F3LICENSEBY). |
F3CERTLICENSE |
ELS F3 |
Dropped |
Is
it a certification, license, or both? |
Dropped. |
Deemed by the TRP to be relatively less important than the licensure/certification question added above (F3LICENSEBY). |
F3BENLICENSE |
ELS F3 |
Dropped |
Which
of the following benefits did you receive or do you plan to
receive from earning this license? |
Dropped. |
Agreed upon by TRP that this question is of relatively less importance, especially given the large amount of time required to administer this form. |
F3JOBLICENSE |
ELS F3 |
Revised |
Were
any of the following required for your job as a [job name]? |
Was
a professional certification or a state or industry license
required for your [current/most recent] job? |
In the FT this question was actually included as part of F3JOBREQ; during the TRP meeting it was suggested that this item would be more clearly understood if included as a distinct question in the set of licensure/certification questions (see also row for F3JOBREQ). |
F3MLTLOC |
ELS F3 |
Dropped |
[Has your military service been/Was your military service]… 1=in the U.S. 2=outside the U.S., or 3=both? |
Dropped. |
Deemed to be of relatively little importance by TRP. |
F3MLTCURBRCH |
ELS F3 |
Dropped |
In
which branch are you currently serving? |
Dropped. |
Deemed to be of relatively little importance by TRP. |
F3MLTSRVTIME |
ELS F3 |
Revised |
What is the total amount of time (in years and months) that you have ever served on active duty? |
Have
you ever served on active duty? |
To lessen respondent burden, TRP suggested that simply knowing whether respondent has ever been on active duty is sufficient. |
F3MLTCMBT |
ELS F3 |
Revised |
What is the total amount of time (in years and months) that you have ever served in a combat zone? |
Have
you ever served in a combat zone? |
To lessen respondent burden, TRP suggested that simply knowing whether respondent has ever served in a combat zone is sufficient. |
F3LOOKJOB |
B&B:93/03 |
Added |
Not asked. |
Are
you currently available and looking for [full-time]
work? |
Added to help researchers distinguish the unemployed from the discouraged worker; B&B:93/03 wording is “During that time, were you available and looking for work?” (question was asked for each out-of-work spell reported by the respondent). |
F3NUMJOB |
ELS F3 |
Revised |
[Including your military service, how/How] many full-time jobs for pay and how many part-time jobs for pay have you held since January 2005? |
[Including
your military service, how/How] many employers have you had since
January 2006? |
TRP agreed that knowing number of employers is more important than knowing number of jobs. The new wording better expresses the intent of the question, which is to assess quality of employment with a given employer. |
F3TWOJOBS |
ELS F3 |
Dropped |
Do you currently have more than one [full-time job/part-time job/military job/job]? |
Dropped. |
Question is superfluous give the inclusion of F3CNUMJOB. |
F3SELFEMP2 |
NLSY |
Added |
Not asked. |
[Are/were]
you… |
Added to help verify/substantiate "working for yourself" responses in preceding question (F3SELFEMP). |
F3SUPERVISE |
NSRCG |
Added |
Not asked. |
[Do/Did] you supervise the work of others as part of your [job name]? (Answer
“yes” if you recommended or initiated personnel
actions such as hiring, firing, evaluating, or promoting others.
Teachers should not count students.) |
Added (per TRP recommendations) to help gauge job autonomy and/or job responsibility; actual question wording in the National Survey of Recent College Graduates is: “Did you supervise the work of others as part of the principal job you held during the week of October 1, 2008? (Mark “yes” if you recommended or initiated personnel actions such as hiring, firing, evaluating, or promoting others. Teachers should not count students.)”. |
F3JOBBEN |
ELS F3 |
Revised |
Did
your most recent employer offer any of the following benefits? |
[Does
your employer for you [job name] offer / Does your current
employer offer / Did your most recent employer offer] medical
insurance or health insurance such as dental or vision? |
"Life insurance" and "Retirement" items were considered to be of relatively less importance by the TRP, and so the question has been revised to simply ask about "medical insurance". |
F3JOBREQ |
ELS F3 |
Revised |
[Are/Were]
any of the following required for your [job name]? |
[Are/Were]
any of the following required for your [job name]? |
To provide further clarity, moved "an industry certification or occupational license" to the certification/licensure line of questions (see also F3JOBLICENSE); to fill a substantive void, added an item for "A graduate degree". |
F3JOBFIELD |
ELS
F3 |
Revised |
Would
you say your [job name] [is/was] related to the major or field of
study you had when you were last enrolled in college? |
How
closely related [is/was] [job name] to the major or field of
study you had when you were last enrolled in college? |
So as to collect more detailed information, revised question from a Yes/No question to a 3-response question; revised wording is based on the following NPSAS:93 question: "How closely was the job related to your major or planned area of study at that time?” (Closely related / Somewhat related / Not related) |
F3EVERFIELD |
ELS
F3 |
Added |
Not asked. |
Have
you ever had a job that was closely related to the major or field
of study you had when you were last enrolled in college? |
See also row directly above for F3JOBFIELD; this question is to be administered to respondents whose current/most recent job is only "somewhat related" or "not related" to their field of study when last enrolled in college. |
F3JOBDETL |
ELS F3 |
Dropped |
Please indicate the extent to which you agree or disagree with each of the following statements with respect to your job as a [job name]: • You are confident that you can perform your job exceptionally well (in other words, much better than average) • You are certain that you can solve big problems that occur at work • You are confident that you can reach the larger goals you set for yourself at work • You are certain that you can do your work well despite time pressures • You are confident that you can do your work well even when you need to juggle work with non-work responsibilities (for example, in your family or community) |
Dropped. |
This question/these items (which make up the Social Cognitive Career Theory construct of "occupational self-efficacy") were newly created for ELS F3 FT. After analyzing field test data, Bob Lent (the items' creator) noted that the data collected via this question/these items were very highly skewed and kurtotic, and recommended that they be dropped. |
F3JOBREMAIN |
ELS F3 |
Dropped |
Your
job as a [job name] allows you to... |
Dropped. |
This question/these items (which make up the Social Cognitive Career Theory construct of "occupation outcomes expectations") were newly created for ELS F3 FT. After factor-analyzing field test data, Bob Lent (the items' creator) noted that the data collected via this question/these items tended to load on the same underlying factor as two other SCCT item sets (i.e. "occupational interests" and "occupational satisfaction"). Given this, Dr. Lent's recommendation was that the "outcome expectations" and "occupational interests" items be dropped (see also F3JOBSTATE below). |
F3JOBSTATE |
ELS F3 |
Revised |
Please
indicate the extent to which you agree or disagree with each of
the following statements about your job as a [job name]: |
Please
indicate the extent to which you agree or disagree with each of
the following statements about your job as a [job name]:
|
After factor-analyzing field test data, Bob Lent (these items' creator) noted that the data collected via the first three items on this form (which make up the Social Cognitive Career Theory construct of "occupational interests") tended to load on the same underlying factor as two other SCCT item sets (i.e. "occupational interests" and "occupational satisfaction"). Given this, Dr. Lent's recommendation was that the "outcome expectations" and "occupational interests" items be dropped (see also F3JOBREMAIN above). |
F3JTRAIN |
ELS F3 |
Revised |
Now we would like to ask you about any formal job training you have received from an employer. Think about the skills that are needed for doing your [job name].In the last 12 months, have you participated in a formal training program offered by an employer or a union that helped you learn or improve the skills needed to do your job?(Include training from past as well as current employers if received in the last 12 months.) (Yes/No)
Before we continue to the next question, we would like to know if you had any difficulty understanding or answering this question so that we may improve it for future surveys. Did you have any difficulty such as not understanding question wording, being uncertain of the meaning of certain terms, or not having the information needed to answer the question? (Yes/No)
If yes, please describe any difficulty you had. Please be as specific as possible. |
Now we would like to ask you about any formal job training you have received from an employer. Think about the skills that are needed for doing your [job name].In the last 12 months, have you participated in a formal training program offered by an employer or a union that helped you learn or improve the skills needed to do your job?(Include training from past as well as current employers if received in the last 12 months.) (Yes/No) |
Removed the "did you have any difficulty answering" follow-up question, which was only intended for eliciting information which might potentially improve questionnaire for FS. |
F3TRAINLEN |
ELS F3 |
Dropped |
About
how long did that training program last? |
Dropped. |
TRP deemed this question of little importance relative to simply knowing whether the respondent has participated in any employer-provided training (F3JTRAIN). |
F3TRAINHRS |
ELS F3 |
Dropped |
On average, about how many hours per [day/week/month] were spent on this training? |
Dropped. |
TRP deemed this question of little importance relative to simply knowing whether the respondent has participated in any employer-provided training (F3JTRAIN). |
F3UNEMPEVR |
ELS F3 |
Revised |
Since
January 2008, have you ever been unemployed, that is, not
employed but seeking employment? |
Since
January 2009, have you ever been unemployed for a period of one
month or more, that is, not employed but seeking
employment? |
Added qualifier "for a period of one month or more" to the question wording in an attempt to weed out transitional unemployment and discouraged workers. |
F3UNEMPFREQ |
ELS F3 |
Revised |
Since January 2008, approximately how many times have you been unemployed (not employed but seeking employment)? |
Since January 2009, approximately how many times have you been unemployed for a period of one month or more (not employed but seeking employment)? |
Added qualifier "for a period of one month or more" to the question wording in an attempt to weed out transitional unemployment and discouraged workers. |
F3UNEMPCOMP |
B&B:93/03 |
Added |
Not asked. |
Since
January 2009, have you ever received unemployment compensation,
applied for unemployment compensation, or been eligible to
receive unemployment compensation? |
The TRP recommended adding a question asking whether respondent had received, applied for, or been eligible for unemployment compensation. As with the newly-added item F3LOOKJOB, this unemployment question helps distinguish the unemployed from discouraged worker, but covering a span of persistent high unemployment in the country. Actual wording from B&B:93/03 is “Since 1997, have you ever received unemployment compensation while you were unemployed and looking for work?” |
F3EMPBAR |
ELS F3 |
Revised |
Since
January 2005, have any of the following interfered with your
career plans? |
Since
January 2009, have any of the following interfered with your
career plans? |
The
TRP recommended that two items ("Taking time off to be a
homemaker or raise children" and "Caring for a sick
relative") be removed because they could be considered more
of a voluntary decision as opposed to a "barrier". |
F3OCC30ED |
ELS
F3 |
Revised |
How
much education do you think you need to get the job you expect or
plan to have when you are 30 years old? |
How
much education do you think you need to get the job you expect or
plan to have when you are 30 years old? |
Revised set of response options to be consistent with other "credential type" questions used in the ELS F3 FS questionnaire (new response options are borrowed from NPSAS:12). |
F3MARSTAT F3PARTNER |
ELS F3 |
Revised |
F3MARSTAT: What is your current marital status? 1=Single, never married 2=Married 3=Divorced 4=Separated 5=Widowed 6=Not married, but living with a significant other in a marriage-like relationship |
F3MARSTAT: What is your current marital status? 1=Single, never married 2=Married 3=Divorced 4=Separated 5=Widowed
F3MARPARTNER: Are
you living with a significant other in a marriage-like
relationship? |
Since FT response options were not mutually exclusive (e.g. someone could be both divorced and living with a significant other), the last response option has been removed and ”whether the respondent is living with a significant other” is asked as a separate question. |
F3MARPARTNER |
ELS F3 |
Dropped |
Have you ever been legally married? |
Dropped. |
This question was previously asked only of FT respondents who indicated their current marital status was "living with a significant other"; given the changes described for F3MARSTAT and F3PARTNER. |
F3SPOUSEED |
ELS
F3 |
Revised |
What
is the highest level of education your spouse has
completed? |
What
is the highest level of education your [spouse/partner] has
completed? |
Revised set of response options to be consistent with other "credential type" questions used in the ELS F3 FS questionnaire (new response options are borrowed from NPSAS:12). |
F3MAREND |
ELS F3 |
Dropped |
How
did your [first/second/third/etc] marriage end? |
Dropped. |
Deemed to be of relatively little importance by the TRP. |
F3BIOCHPART |
ELS F3 |
Revised |
At the time of your [first/second/third/etc.] biological child’s birth, were you married to or partnered with your child’s [father/mother/other parent]? (Yes/No)
Before we continue to the next question, we would like to know if you had any difficulty understanding or answering this question so that we may improve it for future surveys. Did you have any difficulty such as not understanding question wording, being uncertain of the meaning of certain terms, or not having the information needed to answer the question? (Yes/No)
If yes, please describe any difficulty you had. Please be as specific as possible. |
At the time of your [first/second/third/etc.] biological child’s birth, were you married to your child’s [father/mother/other parent]? (Yes/No) |
Per TRP recommendation, removed "or partnered with" from question wording since existing research indicates not being married (as opposed to not being married or partnered) is the primary distinction in determining whether having a child might interfere with other education/employment-related activities. Also removed the "did you have any difficulty answering" follow-up question, which was only intended for eliciting information which might potentially improve questionnaire for FS. |
F3RESZIP |
ELS F3 |
Revised |
Before completing the interview, we would like to collect some information to help in processing your $XX check. (Please update/verify your name below; please also provide a phone number and an address to which you would like your check mailed.) |
Now we’d like to collect some information about your current living arrangements. What is the 5-digit ZIP code for your current residence? |
Though the FT instrument did in fact collect respondent’s ZIP-code in the course of collecting an address to which the incentive check should be mailed, that “incentive ZIP code” (1) may differ from the respondent’s residential zip code, and (2) is provided by the respondent not as a deliverable data element per se, but as information necessary for them to receive their incentive payment. In hopes of including the respondent’s “residential zip code” on the restricted-use datafile (such that researchers are better able to track outcomes, e.g. movement for employment and post-secondary education, intergenerational wealth, and calculating SES) we would like to include a survey item explicitly asking for the respondent’s residential ZIP code. Please note that the wording shown in the “Old Wording” column will still be used at the end of the F3 questionnaire such that an incentive address is obtained. |
F3PARHOME |
ELS F3 |
Revised |
Do
you... Before
we continue to the next question, we would like to know if you
had any difficulty understanding or answering this question so
that we may improve it for future surveys. Did you have any
difficulty such as not understanding question wording, being
uncertain of the meaning of certain terms or response choices, or
not having the information needed to answer the
question? If yes, please describe any difficulty you had. Please be as specific as possible. |
Do
you... |
Removed the "did you have any difficulty answering" follow-up question, which was only intended for eliciting information which might potentially improve questionnaire for FS. |
F3CONTRIBUTE |
AddHealth |
Added |
Not asked. |
Do
you contribute to the cost of your room and board by paying money
to your [parent(s)/guardian(s)], paying certain household bills,
or buying things such as groceries for the household? |
Added per TRP recommendations to address gap in questionnaire regarding inter-generational transfer of wealth; actual question wording from AddHealth survey is: “During the past 12 months, have you contributed toward the cost of your room and board by paying money to another household member, paying certain household bills, or buying things—such as groceries—for the household?” |
F3DPNDNT F3DEPENDENT |
ELS F3 |
Revised |
F3DEPENDENT: Now
we would like to get some information about your current
dependents. Excluding [yourself/you and your spouse/you and your
partner], how many of each of the following types of dependents
do you currently support? *Number of dependent children (less than age 18) *Number of dependent adults (age 18 and older) |
F3DPNDNT: Now we would like to get some information about current dependents. Excluding [you/you and your spouse/you and your partner], do you currently support any dependents?(A dependent is a person for whom you pay at least half their expenses, such as food, shelter, clothing, health care, and schooling. This may include your children, parents, or others. Note that a dependent does not have to live with you.) (Yes/No)
F3DEPENDENT: Excluding
[yourself/you and your spouse/you and your partner], how many of
each of the following types of dependents do you currently
support? |
Since the majority of FT respondents indicated 0 respondents when answering F3DEPENDENT, added this simple yes/no gate question (F3DPNDNT) in hopes of shortening the average interview length; also, to improve clarity, revised item wording to read "Number of dependents less than age 18" (as opposed to "Number of dependent children") and "Number of dependents age 18 or older" (as opposed to "Number of dependent adults"). |
F3EARN2011CAT |
ELS F3 |
Revised |
We
understand that you may not be able to provide an exact number
for your job earnings. However, it would be extremely helpful if
you would indicate which of the following ranges best estimates
how much you earned from employment prior to taxes and deductions
in calendar year 2010. Please include all wages, salaries, net
income from a business or farm, commissions, and tips you earned
in 2010. |
We
understand that you may not be able to provide an exact number
for your job earnings. However, it would be extremely helpful if
you would indicate which of the following ranges best estimates
how much you earned from employment prior to taxes and deductions
in calendar year 2011. Please include all wages, salaries, net
income from a business or farm, commissions, and tips you earned
in 2011. |
Revised response categories based on distribution of FT responses to F3EARN2010; also revised reference period to be more appropriate for FS sample. |
F3SPEARNCAT |
ELS F3 |
Revised |
We understand that you may not be able to provide an exact number for your [spouse’s/partner’s] job earnings. However, it would be extremely helpful if you would indicate which of the following ranges best estimates how much your [spouse/partner] earned from employment prior to taxes and deductions in calendar year 2011. Please include all wages, salaries, net income from a business or farm, commissions, and tips they earned in 2011. 1=Less than $1,000 2=$1,000-$2,499 3=$2,500-$4,999 4=$5,000-$9,999 5=$10,000-$14,999 6=$15,000-$19,999 7=$20,000-$29,999 8=$30,000-$49,999 9=$50,000 and above |
We understand that you may not be able to provide an exact number for your [spouse’s/partner’s] job earnings. However, it would be extremely helpful if you would indicate which of the following ranges best estimates how much your [spouse/partner] earned from employment prior to taxes and deductions in calendar year 2011. Please include all wages, salaries, net income from a business or farm, commissions, and tips they earned in 2011. 1=Less than $1,000 2=$1,000-$2,499 3=$2,500-$4,999 4=$5,000-$9,999 5=$10,000-$14,999 6=$15,000-$19,999 7=$20,000-$24,999 8=$25,000-$29,999 9=$30,000-$34,999 10=$35,000-$44,999 11=$45,000-$54,999 12=$55,000-$64,999 13=$65,000 and above |
Revised response categories to be consistent with F3EARN2010. |
F3INCOME |
ELS F3 |
Dropped |
Did
[you/you or your spouse/you or your partner] receive income from
any of the following sources during 2010? |
Dropped. |
The TRP agreed this question should be dropped given (1) the low frequency of "yes" responses to many of these items during the FT; (2) the limited value of the high level of detail elicited by this question; and (3) the large amount of time required to administer this form (especially in the face of a FT interview which was slightly too long). |
F3PUBASSIST |
NPSAS:12 |
Added |
Not asked. |
During
2011, did you [or anyone else in your household] receive any of
the following benefits: Social Security, SNAP (the Food Stamp
Program), TANF (the Temporary Assistance for Needy Families
Program), The Free and Reduced Price School Lunch Program, or WIC
(the Special Supplemental Nutrition Program for Women, Infants,
and Children)? |
Per TRP guidance, whether or not the respondent has received public assistance is of critical importance; thus, this question is added in place of the dropped question in the row above. Actual NPSAS:12 wording is: "Between July 1, 2010 and June 30, 2011, did you [or anyone in your household] receive any of the following benefits? Social Security, SNAP (the Food Stamp Program), TANF (the Temporary Assistance for Needy Families Program), The Free and Reduced Price School Lunch Program, WIC (the Special Supplemental Nutrition Program for Women, Infants, and Children) |
F3PARASSIST |
AddHealth |
Added |
Not asked. |
Have
your parent(s) or guardian(s) given you any money or paid for
anything significant for you during the last 12 months? Do not
include regular birthday or holiday gifts. |
Added per TRP recommendations to address gap in questionnaire regarding inter-generational transfer of wealth; actual question wording from AddHealth survey is: “Has {HE/ SHE} given you any money or paid for anything significant for you during the past 12 months? Don’t include regular birthday or holiday gifts." |
F3OTHRSRC |
ELS F3 |
Revised |
Without considering any 2010 earnings from employment, approximately how much did [you / you and your spouse / you and your partner] receive from other sources of income in 2010? (These sources might include investments, unemployment compensation, alimony or child support, family members, or disability payments.) |
Without considering any 2011 earnings from employment, approximately how much did [you / you and your spouse / you and your partner] receive from sources of income other than employment in 2011? (If none, please enter ‘0’. Sources of income other than employment might include investments, unemployment compensation, alimony or child support, family members, or disability payments.) |
Made slight revisions to question wording to improve clarity given that F3INCOME has been dropped. |
F3RFUND F3RPLANSIN F3RPLANMAR |
ELS F3 |
Revised |
F3RFUND: Now
we would like to ask you some questions about any pension or
retirement savings. Many employers and unions have pensions or
retirement plans, some provide tax-deferred plans such as thrift,
savings, 401 K's, profit sharing, or stock ownership plans.
Additionally, individuals can provide for their own retirement
with IRA or Keogh plans. Do [you/you or your spouse/you or your
partner] have any savings in these types of plans? |
F3RPLANSIN: Now
I would like to ask you some questions about any pension or
retirement savings. Many employers and unions sponsor pensions or
retirement plans, some provide tax-deferred plans such as 401
K's, profit sharing or stock ownership plans. Other examples
include thrift savings plans, 403B plans, 457 plans, IRA-SEP
plans, IRA-SIMPLE plans, and other IRA plans. Do you have any
savings in these types of plans?
F3RPLANMAR: Now I would like to ask you some questions about any pension or retirement savings. Many employers and unions sponsor pensions or retirement plans, some provide tax-deferred plans such as 401 K's, profit sharing or stock ownership plans. Other examples include thrift savings plans, 403B plans, 457 plans, IRA-SEP plans, IRA-SIMPLE plans, and other IRA plans. Response options: yes/no for each item below *Do you and your [spouse/partner] have any of these plans jointly? *Do you have any of these plans on your own separate from your [spouse/partner]? *Does your [spouse/partner] have any of these plans separate from you? |
Given
that response options 1 and 4 for F3RFUND are not mutually
exclusive, this question is being replaced with F3RPLANSIN (to be
administered to non-married and non-partnered respondents) and
F3RPLANMAR (to be administered to married or partnered
respondents). Please also note the examples listed in the
question wording for F3RPLANSIN and F3RPLANMAR have been updated
to reflect the question wording used in the latest round of NLSY;
the question appears in NLSY round 13 as follows: "Now I
would like to ask you some questions about any pension or
retirement savings. Many employers and unions sponsor pensions or
retirement plans, some provide tax-deferred plans such as 401
K's, profit sharing or stock ownership plans. Do [you/you or your
spouse/you or your partner] have any savings in these types of
plans? |
F3DOMICILE |
ELS F3 |
Revised |
Do
you... |
Do
you... 3=have some other arrangement? |
Slightly revised wording for response option #2 to improve clarity. |
F3ALLDEBT |
ELS F3 |
Revised |
Now,
think about your debts[, besides any mortgage on your home]. How
much do you[ and others in your household] owe altogether? [Do
not include mortgage loans, but include all other debts / Include
all debts,] such as all other loans, credit card debt, medical or
legal bills, etc. |
Now,
think about your debts[, besides any mortgage on your home]. How
much do you[ and your spouse/partner] owe altogether? [Do not
include mortgage loans, but include all other debts,/Include all
debts] such as credit card debt, unpaid student loans, unpaid car
loans, and all other unpaid loans. |
Revised question wording to better distinguish between "debt" and "bills", and to better distinguish between respondent/spouse/partner debt and family debt (e.g. parental debt). |
F3FINSTRESS |
ELS F3 cognitive tests |
Added |
Not asked. |
Many young adults experience financial problems. How much stress have you felt in meeting your financial obligations during the past year? 1=Not at all stressful 2 3=Moderately stressful 4 5=Extremely stressful |
Added per TRP recommendations; this item performed well during cognitive testing. |
F3VOTE |
ELS F3 |
Revised |
Are
you currently registered to vote? |
Now
we have some questions on voting. Are you currently registered
to vote? |
Added transition phrase to improve questionnaire flow. |
F3VOTELOCAL |
ELS F3 |
Revised |
In the last 2 years, have you voted in any local, state, or national election? |
Did you vote in any local, state, or national election during 2009, 2010, or 2011? |
Since the 2012 presidential election will take place during the ELS F3 FS data collection window, question wording has been revised to better distinguish between presidential elections and non-presidential "national elections". |
F3VOLORG |
ELS F3 and ELS F3 cognitive tests |
Revised |
Which
of the following types of organizations have you been involved
with in your unpaid volunteer or community service work during
the past two years? |
Which
of the following types of organizations have you been involved
with in your unpaid volunteer or community service work during
the past two years? |
Revised set of volunteer groups to match those asked about in ELS cognitive tests; this item performed well during cognitive testing. |
F3VALUES |
ELS F3 |
Revised |
How
important is each of the following to you in your life? |
How
important is each of the following to you in your life? |
Revised last two items to use more present-tense language; per TRP recommendations, eliminated two items ("Being successful in your line of work" and "Being able to find steady work") due to high frequency of 'yes' responses; eliminated "Getting away from this area of the country" as it is more relevant to the base year questionnaire (from which these items were drawn). |
F3SSN |
ELS F2 |
Added |
Not asked. |
What is your Social Security number? (Under Title 20 of the General Education Provisions Act, we may collect your Social Security number for the purpose of confirming information abstracted from postsecondary educational records. Giving us your Social Security number is completely voluntary and there is no penalty for not disclosing it.) |
Administered only to those respondents for whom we do not already have a valid Social Security number in hopes of facilitating collection of postsecondary transcripts and/or financial aid records. |
F3REINTSEL |
ELS F3 |
Dropped |
You
have been randomly selected for participation in a quality
control interview. We would like you to return to this web site
in about three weeks to repeat a small number of questions. The
purpose of this second 10-minute interview is to determine how
well our questions collect reliable information. |
Dropped. |
No re-interview for main study. |
December 2011
Education
Longitudinal Study: 2002
(ELS:2002)
Third Follow-up 2012 Full-scale Study
OMB Supporting Statement
Part D
Linkages to Extant Data Sources
OMB# 1850-0652 v.8
National Center for Education Statistics
Institute of Education Sciences
U.S. Department of Education
TABLE OF CONTENTS
Section Page
D.1 Develop Linkages with Extant Data Sources D-2
D.2 File Merge with ED Central Processing System (CPS) D-3
D.3 File Merge with National Student Loan Data System Disbursement D-4
D.4 File Merge with General Educational Development Testing Service D-4
D.5 Processing Extant Data D-4
D.1 Develop Linkages with Extant Data Sources
As part of ELS:2002, linkages will be developed with several existing data sources to supplement the student interview data. Because certain data (for example, specific financial aid amounts and associated dates) can only be accurately obtained from sources other than the student or parent. Through the experience of gathering data for many National Center for Education Statistics (NCES) studies, including previous ELS:2002 data collections, Baccalaureate and Beyond Longitudinal Study (B&B), National Postsecondary Student Aid Study (NPSAS), and Beginning Postsecondary Student (BPS) study, considerable knowledge has been gained performing file merges with external sources of data, including Department of Education’s (ED) Central Processing (CPS) for Free Application for Federal Student Aid (FAFSA) data, the National Student Loan Data System (NSLDS), and the General Educational Development (GED) Testing Service. For this study, we propose to merge ELS:2002 files with the following datasets: GED, CPS, and NSLDS.
The Family Educational Rights and Privacy Act (FERPA), (34 CFR Part 99) allows the disclosure of information without prior consent for the purposes of ELS:2002 according to the following excerpts: 99.31 asks “Under what conditions is prior consent not required to disclose information?” and explains in 99.31 (a) that an educational agency or institution may disclose personally identifiable information from an education record of a student without the consent required by 99.30 if the disclosure meets one or more specific conditions. ELS:2002 collection falls under Sec. 99.31 (a)( 3). The disclosure is, subject to the requirements of Sec. 99.35, to authorized representatives of--
The Comptroller General of the United States;
The Attorney General of the United States;
The Secretary; or
State and local educational authorities.
ELS:2002 is collecting data under the Secretary’s authority. The personally identifiable information is collected from student record systems with adherence to the security protocol detailed in 99.35: “What conditions apply to disclosure of information for Federal or State program purposes?”
(a)(1) Authorized representatives of the officials or agencies headed by officials listed in Sec. 99.31(a)(3) may have access to education records in connection with an audit or evaluation of Federal or State supported education programs, or for the enforcement of or compliance with Federal legal requirements that relate to those programs.
(2) Authority for an agency or official listed in Sec. 99.31(a)(3) to conduct an audit, evaluation, or compliance or enforcement activity is not conferred by the Act or this part and must be established under other Federal, State, or local authority.
(b) Information that is collected under paragraph (a) of this section must:
(1) Be protected in a manner that does not permit personal identification of individuals by anyone other than the officials or agencies headed by officials referred to in paragraph (a) of this section, except that those officials and agencies may make further disclosures of personally identifiable information from education records on behalf of the educational agency or institution in accordance with the requirements of Sec. 99.33(b); and
(2) Be destroyed when no longer needed for the purposes listed in paragraph (a) of this section.
(c) Paragraph (b) of this section does not apply if:
(1) The parent or eligible student has given written consent for the disclosure under Sec. 99.30; or
(2) The collection of personally identifiable information is specifically authorized by Federal law.
Secure Data Transfers. NCES has set up a secure data transfer system, using their NCES member site with Secure Sockets Layer (SSL) technology, described above. The study contractor, RTI, will use this electronic system for submitting data containing potentially identifying information (such as SSNs, names, and dates of birth of our sample members) along with their survey ID (not the same ID that is available on the restricted-use data). Before being transmitted, files will be encrypted using FIPS 140-2 validated encryption tools. RTI will receive data from the NCES system as well. The system requires that both parties to the transfer be registered users of the NCES Members Site and that their Members Site privileges be set to allow use of the secure data transfer service as described above. This process will be used for all file matching procedures described below, except in instances when the vendor already has a secure data transfer system in place.
D.2 File Merge with ED Central Processing System (CPS)
RTI will perform file merges with the CPS data containing federal student aid application information. The merge with CPS can occur at any time for any number of cases, provided that the case has an apparently valid SSN associated with it. RTI sends a file to CPS and receives in return a large data file containing all students who applied for federal aid. Programs and procedures are in place to prepare and submit files according to rigorous CPS standards. Similarly, programs and procedures are in place to receive and process data obtained from CPS.
RTI will electronically upload a file on the FAFSA secure web-site for matching which contains SSN and the first 2 letters of the sample member’s last name (but no other information). Access to the site for the upload is restricted to authorized users who are registered and provide identification/authentication information (SSN, DOB, and personal identification number [PIN]) to the FAFSA data site. The file is retrieved by the Central Processing System or CPS (the FAFSA contractor data system) for linkage. The linked file, containing student aid applications for matched records, is then made available to RTI only through a secure connection (EdConnect) which requires username and password. All CPS files will be processed, edited, and documented for inclusion on the analytic data files.
D.3 File Merge with National Student Loan Data System Disbursement
RTI will also conduct a file merge with the NSLDS to collect federal loan and Pell grant data. The resulting file will contain cumulative amounts for each student’s entire postsecondary education enrollment. Programs exist to create the files for the merge and others to read the received data. All matching processes are initiated by RTI staff providing a file with one record per sample member. File transfers will use the NCES secure data transfer system described above.
D.4 File Merge with General Educational Development Testing Service
RTI will also conduct a file merge with the GED testing service, as was done in ELS:2002/06 to obtain GED test dates and results. This will extend the coverage for the ELS:2002 sample regarding attempts for high school equivalency credentialing. File transfers will use the NCES secure data transfer system described above.
D.5 Processing Extant Data
The data from all of these sources, as allowed by the vendor, will be delivered in the restricted-use data files and will be useful for creating derived variables. The variables – both direct and derived – will be documented thoroughly for the data files.
December 2011
Education
Longitudinal Study: 2002
(ELS:2002)
Third Follow-up 2012 Full-scale Study
OMB Supporting Statement
Part E
Response Propensity Modeling: Field Test Experiment and Full-scale Study Plans
OMB# 1850-0652 v. 8
National Center for Education Statistics
Institute of Education Sciences
U.S. Department of Education
TABLE OF CONTENTS
Section Page
E.1 Background of Field Test Experiment Approach E-2
E.2 Treatment for Low Propensity Cases E-4
E.3 Propensity Modeling Conclusions E-7
E.4 Plans for Full Scale Study E-7
E.5 Analysis from the Second Follow-up Full Scale Study E-9
References E-11
Exhibits
Number Page
Exhibit E-1. Distribution of Low Propensity Cases by Prior Response Status E-3
Exhibit E-2. Distribution
of Response Propensities by Sample Member and High School
Characteristics E-4
Exhibit E-3. ELS:2002 FT Treatment Schedule E-4
Exhibit E-4. Response Rates by Propensity and Experimental Group E-5
Exhibit E-5. Estimates of Select Survey Variables By Propensity Group (p<.05) E-5
Exhibit E-6. Unit-level Bias Analyses E-7
Exhibit E-7 Average Mahalanobis Distance Score by Month of Data Collection E-10
E.1 Background of Field Test Experiment Approach
Under a contract with NCES, modeled on the Responsive Design methodologies developed by Groves (Groves and Heeringa, 2006), RTI is currently developing new approaches to improve survey outcomes that incorporate different responsive and adaptive features. After sample members with the lowest response propensities are empirically identified, they are targeted with interventions in an attempt to encourage participation. Inclusion of low propensity cases has the potential to ameliorate increases in standard errors arising from non-response adjustments. To inform main study operations, design, and content, ELS:2002 includes a field test sample. To estimate a case’s response propensity prior to the start of the ELS:2002 third follow-up field test, a sample member’s eventual response status in the ELS:2002 second follow-up field test was predicted. A logistic regression model was fitted with the sample member’s ELS:2002 second follow-up field test response status as the dependent variable. As independent variables, a range of information known for all respondents and nonrespondents from each prior wave of the longitudinal field test, including information from batch tracing activities were examined for significance. The following variables were considered as predictors of a field test sample member’s second follow-up field test response outcome: base year response status, first follow-up response status, whether the respondent ever refused, whether the respondent has ever scheduled an appointment, whether the respondent was classified as hard to reach, the number of calls made to the respondent in the second follow-up, high school completion status, parental level of education, high school type, urbanicity, dropout status, and the sample member’s postsecondary aspirations.
No information about the race, gender, or any other demographic characteristics of the sample members was used for prediction.
Predicted probabilities derived from the logistic regression model were used to get an estimate of a case’s response propensity for the third follow-up field test. Cases were split into two groups of equal size. Field test sample members above the median response propensity were classified as high propensity, and those below the median as low propensity. In total, 528 cases were classified as high propensity and 527 as low propensity. For the implementation of the experiment, the 527 low propensity cases were randomly split into experimental and control groups. The goal was to examine how well low propensity cases using prior wave data can be predicted and how these should be treated to change their response propensity.
Since low propensity cases assigned to the experimental group received treatment, of interest was how those cases were distributed according to their prior response status. Exhibit E-1 shows the distribution.
Exhibit E-1. Distribution of Low Propensity Cases by Prior Response Status
|
BY Response Status |
First Follow-up Response Status |
Second Follow-up Response Status |
|||
|
% Respondent |
% Nonrespondent |
% Respondent |
% Nonrespondent |
% Respondent |
% Nonrespondent |
Third Follow-up FT Low Propensity Cases |
85% (447) |
15% (80) |
78% (412) |
22% (115) |
55% (292) |
45% (235) |
Third Follow-up FT High Propensity Cases |
96% (507) |
4% (21) |
100% (528) |
0% (0) |
92% (488) |
8% (40) |
Note: Actual counts of cases in parentheses.
As shown in Exhibit E-1, the low propensity cases consisted of both respondents and nonrespondents in all prior waves of ELS:2002. Also, high propensity cases were not limited to second follow-up respondents. A number of nonrespondents were classified as high propensity. This suggests that for ELS:2002 prior round response status, while important, may not be sufficient as a predictor of response outcome in the third follow-up and should not be the sole basis for partitioning cases into propensity categories.
Exhibit E-2 shows the distribution of the case propensities across some demographic characteristics of interest. From the data, it is clear that the demographic distribution of the propensities approximates the distribution in the overall FT sample. There is no obvious skewing across these demographic characteristics.
Exhibit E-2. Distribution of Response Propensities by Sample Member and High School Characteristics
|
Percent (and number) of cases in FT Sample |
Percent (and number) of Cases in High Propensity Category |
Sample Member Characteristics |
|
|
Male |
50.3 (531) |
47.2 (249) |
White |
55.0 (550) |
58.9 (293) |
Black |
18.8 (188) |
17.7 (88) |
Hispanic |
19.4 (194) |
15.7 (78) |
Asian |
6.2 (62) |
7.0 (35) |
High School Characteristics |
|
|
Urban |
40.1 (431) |
38.3 (202) |
Public |
84.3 (889) |
86.5 (457) |
E.2 Treatment for Low Propensity Cases
The basic premise of the response propensity approach is to identify low propensity cases as early as possible and assign to them modified treatments. In theory, treating low propensity cases in the same manner as high propensity cases is inefficient and possibly harmful to overall data quality. The special treatment for ELS:2002 field test low propensity cases was a higher incentive of $45 at the start of data collection (weeks 1-9) and going up to $55 starting at week 10. High propensity and control group cases were offered $25 until week 10 of data collection, after which the incentive increased to $35. Exhibit E-3 outlines the timing and levels of the different incentives.
Exhibit E-3. ELS:2002 FT Treatment Schedule
|
High Response Propensity |
Low Response Propensity |
|
Week |
All High Cases |
Control Group |
Experimental Group |
1-9 |
$25 |
$25 |
$45 |
10+ |
$35 |
$35 |
$55 |
Results of the Field Test Response Propensity Experiment
The predictive model developed ahead of the field test data collection effectively predicted the eventual response outcome for sample members. The high propensity group’s response rate (67.4%) was significantly higher than that of the low propensity control group (45.4%). This difference was statistically significant (χ2 = 34.9; p < .0001). In examining the effect of the higher incentive treatment for low propensity cases, a numerical difference in participation (51.6% for treatment cases v. 45.4%) was observed, however, the difference was not statistically significant. The increased incentives corresponded to a 6 percent observed increase in response rate for the low-propensity cases. However, the small ELS:2002 field test sample size and the brevity of the data collection period may have contributed to the inability to detect a significant difference in the results.
Exhibit E-4. Response Rates by Propensity and Experimental Group
|
Percentage Respondents |
Percentage Nonrespondents |
High Propensity |
67.4 |
32.6 |
Low Propensity (Control) |
45.4 |
54.6 |
Experimental group |
51.6 |
48.4 |
Control group |
45.4 |
54.6 |
One of the analytical goals of the field test was to determine if high and low propensity cases in ELS:2002 answer critical survey questions differently. Following is a list of select survey variables showing significant differences between propensity groups (p<.05). Their estimates can be seen in Exhibit E-5.
Whether respondent has earned a regular high school diploma
Whether respondent has attended a postsecondary institution
Highest level of postsecondary (PS) school attended
PS attainment (specifically, PS attendance with no credential, and Bachelors degree)
Educational expectations (specifically, 1- or 2-year degree, Masters, and PhD)
Current marital status (specifically, partnered/living with a significant other)
Whether respondent is currently registered to vote
Exhibit E-5. Estimates of Select Survey Variables By Propensity Group (p<.05)
Group |
Low Propensity |
High Propensity |
|
|
Has a regular HS diploma |
91.0 |
95.8 |
|
No PS attendance |
12.4 |
6.2 |
|
Highest PS institution attended is a 4-year |
53.7 |
64.3 |
|
PS attendance but no credential |
44.2 |
29.4 |
|
Highest PS credential earned is a Bachelor’s |
26.6 |
39.9 |
|
Highest level of education expected: complete 1 or 2 year program |
17.5 |
10.9 |
|
Highest level of education expected: Master’s |
22.8 |
30.5 |
|
Highest level of education expected: PhD, professional doctorate |
8.9 |
16.1 |
|
Current marital status is living with significant other |
13.7 |
7.5 |
|
Currently registered to vote |
79.8 |
89.4 |
Other variables showing significant differences between propensity groups (p<.05) include
Currently taking courses at voc/tech/trade school
Currently working and attending PS school
Debt/asset ratio is in highest quartile
Other variables with differences between propensity groups approaching significance (p<.1) include
If sold assets and paid debt: break even
Debt/asset ratio is in middle two quartiles
Voted in presidential election
Voted in state or local election
Variables showing no significant differences between propensity groups include
Currently working at full-time job
Currently working at part-time job
Currently taking courses at 2- or 4- year college
Not currently working & not currently attending PS school
Currently working but not attending PS school
Currently attending PS school but not working
Timing of high school credential
Taken out any education loans
Amount owed on education loans
Number of hours working per week
Ever unemployed since January 2008
Total number of months unemployed
Ever in the military
Whether respondent has biological child
Frequency of volunteering
Mean relative bias was also examined (refer to Exhibit E-6). It is calculated by finding the simple average of a set of relative biases. For the field test analysis, 23 relative biases were calculated from seven variables (sex, race, ever dropout status, high school completion status, base-year school type, base-year urbanicity, and base-year school FIPS state code), and the mean relative bias was calculated by calculating their simple average.
Exhibit E-6. Unit-level Bias Analyses
Group |
Mean relative bias |
All |
10.4 |
All with low propensity cases treated as nonrespondents |
17.6 |
Propensity group |
|
High propensity |
11.1 |
Low propensity |
8.4 |
Experiment condition |
|
Low propensity control |
13.7 |
Low propensity treatment |
10.3 |
Bias is defined as the weighted absolute difference between an estimate calculated using respondents only and an estimate calculated using nonrespondents only. The weight for a bias is defined as the nonresponse rate associated with the bias calculation. A relative bias is calculated by dividing the bias by the estimate calculated using both respondents and nonrespondents. In reviewing the mean relative bias shown in Exhibit E-6, it appears that including low propensity cases in the dataset may help reduce bias (even if slightly), and that the higher incentive for the low propensity experimental cases may have lowered the bias relative to the low propensity control group.
E.3 Propensity Modeling Conclusions
The response propensity model successfully predicted response outcome. The inclusion of low propensity cases showed an apparent reduction in unit-level biases. Including more low-propensity cases in the data may reduce bias and may help improve final estimates since low-propensity cases appear to be different in terms of their survey responses. A higher incentive amount produced an observed higher response rate for low propensity cases. It is not known how the experiment would have concluded if the field test data collection had continued two additional months to its scheduled end. The data collection ended two months early because the interview yield goal was met. If data collection had continued, more cases would have been traced and located, and more cases would have been successfully reached and interviewed. Please refer to B.3 for full scale study recommendations.
E.4 Plans for Full Scale Study
While the propensity modeling approach yielded some positive outcomes, for ELS:2002/12 full scale study, we propose to use a Mahalanobis distance function calculated at three points in data collection to identify nonrespondent cases which are most unlike existing respondents. The goal remains to identify cases during data collection that exhibit large differences from the mean respondent case. Doing so provides an opportunity to target (i.e. with incentives) during nonresponse follow-up, those cases most likely to reduce nonresponse bias. The distance function approach may provide a better opportunity to target cases that exhibit differences across a larger number of variables, rather than relying on propensity to respond.
The following variables will be used to calculate the Mahalanobis distance:
Substantive data already collected/available from base year, first and second follow-up: enrollment status, parent’s education, high school completion status, test score quartile, income quartile, native English speaker, school control, school locale, postsecondary enrollment, current work status, and whether the case was in school in grade 12.
Paradata from current and prior rounds: response status from prior rounds, whether sample member contacted the help desk, whether sample member logged in but did not complete the questionnaire, number of contact attempts in the early data collection period, whether sample member made an appointment, and whether sample member told interviewer they would do the web interview. The Mahalanobis function incorporates a measure of the likelihood of ultimate response among current nonrespondents. Therefore, paradata are very important to include with the substantive data for factoring into the case-selection process thereby optimizing the selection itself.
Mahalanobis distance can be defined as the distance between a case and some group centroid. Therefore, cases with larger distance scores can be thought of as cases demonstrating large differences from the group. That is, these cases would be characterized by differences, for example, in their enrollment status, parent’s education, high school completion status, etc. Identifying these cases and presenting the specifically-targeted nonresponding cases with a higher incentive will in turn attempt to boost their participation and potentially reduce bias in estimates and also improve analytic power through higher sample sizes for these groups of cases of analytic interest.
Cases identified for targeting via their Mahalanobis distance (nonrespondent cases with distance scores furthest from the mean respondent) will receive a $55 incentive – rather than the $25 base incentive – in an attempt to encourage their participation.
We should note that ever-dropout cases are a critical set of cases and will receive $55 from the start of data collection rather than being included in the Mahalanobis distance measure selection process during data collection. The approach with ever-dropouts is based on the particular difficulty gaining cooperation from this important subgroup as demonstrated during the F3 field test and prior rounds of data collection.
We propose to measure the distance functions at three points during data collection. At these points, the cases with the largest distance scores will be offered the $55 incentive while the $25 base incentive will remain intact for all other cases. Other activities that will be conducted to locate and interview targeted cases include performing pre-data collection intensive tracing, pursuing the cases in person with field locator/interviewers, and sending a $5 prepaid incentive near the end of data collection. The case-identification for more-intensive effort will be done at three points during data collection.
Our first target-case-selection point will be right before outbound CATI is scheduled to begin, 4 weeks into data collection.
The second point will be right before the CAPI period begins, 9 weeks into data collection.
The third point will be just prior to the pre-paid incentive period, approximately 8 weeks prior to the end of data collection.
At each juncture, the cases targeted will be those with the largest distance scores but not targeted in the prior phase(s). Also at the third and final point, cases identified for targeting will receive a $5 prepaid incentive in addition to the $55 incentive and other non-monetary activities, listed above. At this late point in data collection, these cases are likely to be the most critical for reducing the final mean distance function scores between respondents and nonrespondents, thus potentially improving analytic power in terms of sample size and reducing bias in estimates. We believe it prudent to be highly strategic with the use of prepaid incentives, which are normally very effective. Again, cases not targeted will continue to be offered the $25 incentive and will not be targeted for the more-intensive non-monetary activities.
Case targeting will be based on distance scores and anticipated yield. Based on the proportion of interviews projected across each of the phases of data collection, the distance scores will be segmented such that the following approximate numbers of cases are to be targeted at each cut point: 915 cases at the first cut point, 760 cases at the second cut point, and 830 at the third cut point. At each of these time points, we will evaluate the Mahalanobis values among the respondents and nonrespondents to identify the target cases.
E.5 Analysis from the Second Follow-up Full Scale Study
We computed Mahalanobis values over time with the second follow-up data to identify variables for the third follow-up and to get a feel for how the metric will be implemented in the third follow-up. Exhibit E-7 shows the Mahalanobis distance function calculated by month of data collection in the ELS:2002 second follow-up main study.
Exhibit E-7 Average Mahalanobis Distance Score by Month of Data Collection
As shown in Exhibit 7, over time in the second follow-up, the distance between respondents and nonrespondents grew slightly larger indicating that these groups may have become more dissimilar as data collection proceeded. The differences in mean distance scores between respondents and nonrespondents were not significant. However, this is illustrative of the potential benefits of carefully monitoring these values over time. If the differences were significant at the end, the risk of bias would have been increased. The goal for the ELS:2002 responsive design in the third follow-up full scale study will be to attempt to minimize the distances between respondents and nonrespondents over the course of data collection and therefore, reduce the risk of nonresponse bias.
References
Groves, R. M., & Heeringa, S. (2006). Responsive design for household surveys: tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society Series A: Statistics in Society, 169(Part 3), 439-457.
Rosen, J. A., Murphy, J. J., Peytchev, A., Riley, S., & Lindblad, M. (2011). The effects of differential interviewer incentives on a field data collection effort. Field Methods, 23, 24–36. (doi:10.1177/1525822X10383390)
Peytchev, A., Riley, S., Rosen, J. A., Murphy, J. J., & Lindblad, M. (2010). Reduction of nonresponse bias in surveys through case prioritization. Survey Research Methods, 4(1), 21–29. http://w4.ub.uni-konstanz.de/srm/article/view/3037.
1 Blumberg, S.J., and Luke, J.V. (December 2009). Wireless Substitution: Early Release of Estimates From the National Health Interview Survey, January–June 2009. National Center for Health Statistics. Available from http://www.cdc.gov/nchs/nhis.htm.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | December 2010 |
| Author | cannada |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |
© 2024 OMB.report | Privacy Policy