Suggestions for the Last ELS Follow-up
Randall J. Olsen
Department of Economics and
Center for
Human Resource Research
Ohio State University
As one thinks of instrumentation for the ELS 2002 survey, it helps to return to first principles and think about the overall purpose and potentially unique contribution of this study, which starts with sophomores in 2002 (and hence about 16 years old) and ends in 2012—the interview in question.
What is unique about the ELS and its kin is the detail on education, especially in terms of what is going on within a particular school and the student’s classroom, and on to cover post-secondary education. Having excellent detail on secondary education has the benefit of enabling a better understanding of post-secondary education. One of the axioms of education research is that no matter when you start studying the child, you will wish you had started earlier. Few studies have the patience and determination to start at birth or before and continue on through college. Those that do, such as the Children of the NLSY, do not have the wealth of data at the school level the ELS and its peer studies do. No one study can do it all.
What is important about the ELS and other studies mounted by NCES is that they can do a far more comprehensive job of focusing on the quality and attributes of education, whereas other studies are forced to content themselves with less demanding measures of the quantity of education even if in some cases these other studies generating better data on the quality of the child’s background and maternal characteristics.
If the issue of school quality is central to the contribution the ELS can make, one way of approaching the design problem is to ask which sorts of questions will provide the greatest resolution on outcomes central to the school quality domain? I will nominate some questions that may help with this in points below.
The area of survey content that is perhaps in the best shape is the way the ELS looks at post-secondary education in terms of applications, acceptances, and post-secondary educational outcomes. Those issues are terribly important and play to the strength of NCES. If we stipulate that NCES has this set of issues well in hand, we can turn to issues related to labor market outcomes and their linkage to school quality, attributes, and what the student learns in school.
One area of inquiry that a coalition of countries in the EU have been pursuing is to measure the extent to which post-secondary schooling (and training) is utilized by graduates. This is an idea that has great face validity. To be sure, utilization and/or usefulness can be rather inchoate descriptors, but allowing for a variety of descriptors is what one would do if such questions were asked. Given this will be the last interview, this would be the wave where more summary concepts should be pursued. After all, this interview will not be preparing the way for an event history to follow.
Here are some sample questions that may need revision, with the line of questioning being conditioned on the highest level of education or training received.
For college graduates and beyond:
Since graduating with your
(fill) degree in (fill field), have you ever worked in a job for
which your degree was a necessary qualification?
If yes: Are you still working in a job for which that degree is a qualification?
If yes: How satisfied are you with the training that degree provided you in terms of preparing you for your job? (5 point satisfaction scale)
If no: Thinking of that same degree, have you ever worked in a job for which your degree may not have been a necessary qualification, but provided skills that were clearly advantageous?
If yes: Are you still working in a job for which that degree was a clear advantage?
If yes: How satisfied are you with the training that degree provided you in terms of preparing you for your job? (5 point satisfaction scale)
If no to the questions on a degree being used:
Since graduation,
have you ever worked in a job for which your degree was a necessary
qualification?
If no: since graduation have you ever worked in a job for which your degree provided skills that were clearly advantageous?
You could also craft parallel questions that cover cases where the highest level of training revealed is vocational training, associate’s degree, apprentice training, adult workforce education, Adult Basic Literacy Education, and so forth.
Turning to the skills issue, if it is not already being done, combining the DOT-O*Net skill characteristics for jobs and cross-matching to area of study may be a fruitful way of generating data on the extent to which education and training match up with jobs held. In this regard, some thought should be given to what to ask about changes in jobs and/or occupations the respondent may have gone through. In the mid-20s, many young workers are starting to change occupations even as they change jobs.
For students who incurred debt for their schooling, the final interview could be used to elicit how that debt is being repaid. That is, ask the R what fraction of the debt has been forgiven by one program or another, what fraction has been paid off by them or their family, and what fraction remains to be paid off.
The NELS is hardly the place to collect balance sheet data, but there a couple of quick ways to determine how the R is doing financially. Most asset accumulation takes place in the form of housing so asking whether R owns or is buying their own home is a good starting point. Getting the value of the home and the size of the mortgage are, in most cases, feasible data collection objectives. Asking the respondent if they have started a retirement fund such as an IRA, 403b, 401k, etc. should likewise be feasible, including how much they have in all such accounts. In the 20’s, stopping there should be sufficiently informative. I would assign this topic a lower priority.
Asking how many full-time jobs the respondent has held since the previous interview will help put a lot of the economic data in context. After all, people change jobs either because they are fired or laid off, or to improve their situation. If we want to understand how effective education has been in improving economic circumstances, we want to measure job changing behavior as this is one of the key pathways through which earnings increase.
I suggest cutting the job satisfaction questions. The fact of job mobility and rates of pay tells us what we need to know. After all, people move to better opportunities. In this regard, the dropped job satisfaction questions could be replaced by questions about the first full-time job after high school (if current job is not the first, else first job after HS if not the current job). In particular, a high priority should go to collecting sufficient information to elicit the hourly rate of pay on that job (see next point). In that way, one can look at earnings growth, which is often very strong early in the career.
On earnings, the sequence in the ELS is suboptimal. I suggest using the protocol in the NLS. We start by asking the respondent:
[Now we would like to ask you a few questions concerning your earnings at (name of employer)] [(READ IF NECESSARY FOR TEACHERS): Please report your earnings (over the number of months that you were (contracted for/paid)./as you were paid over a (9/10) month period).]
For [(this job/current assignment or on-call job/business)] with [(employer name)]([QES-LOOP1 counter]), what is the easiest way for you to report your total earnings before taxes or other deductions: hourly, weekly, annually, or on some other basis?
(READ IF NECESSARY:) We use this information to compare the amount that people earn in different types of jobs.
2627 1 per hour
33 2 per day
425 3 per week
349 4 bi-weekly (every 2 weeks)
43 8 bi-monthly (twice a month)
227 5 per month
1977 6 per year
52 7 other (SPECIFY)
-------
5733
Even though you told me it is easier to report your earnings [time unit text for rate of pay]([QES-LOOP1 counter]), [are/were] you paid at an hourly rate on [(this job/current assignment or on-call job/business)] with [(employer name)]([QES-LOOP1 counter])?
1158 1 Yes (Go To T13750.00)
1846 0 No
-------
3004
(note: some intermediate checks and skips not shown)
This approach maximizes the frequency with which we receive an hourly rate of pay from the respondent, which attenuates measurement error problems that arise when the time unit of rate of pay is other than hourly and one must use reported hours to calculate the hourly rate of pay. This is important as hourly rate of pay is a workhorse variable of labor market analysis.
I would carefully consider removing the questions on smoking and health insurance. Seemingly everyone asks these questions, so the issue here is why the ELS data makes a contribution to understanding these issues as opposed to some other survey (actually, it is probably more accurate to say any other survey as the questions are ubiquitous). These questions are not harmful, but dropping them will create space to questions more tightly linked to the efficacy of the educational system and how education relates to career development. Elsewhere in this memo I give a list of questions for consideration, pretty much all of which would make more sense for inclusion than questions on smoking and health insurance.
In the event health-related questions are desired in the ELS, I would suggest a different set of questions. Specifically, I recommend including a 7-item CES‑D. According to the Partnership for Workplace Mental Health, mental illness causes more days of work loss and work impairment than chronic health conditions such as diabetes, asthma, arthritis, back pain, hypertension, and heart disease. Depression is likely the largest contributor to mental illness. To be sure, the estimate of the importance on mental health comes from an advocacy group, and may need verification and depression is a significant co-morbidity with other non-mental health conditions such as diabetes and heart disease. Nonetheless, depression is surely one of the more serious factors impairing workers. The CES‑D is often criticized for the relatively short reference period it uses for depressive symptomology, but in work currently underway, I use item response theory to analyze CES‑D results over a decade for several thousand women and find less than half the variance in the latent depressivity score is due to randomness specific to the respondent at the time of the interview. The rest of the variance is explained by either a time-invariant fixed effect or an autoregressive error component with a very high serial correlation.
The implication of this stability in underlying depressivity is that a single measure taken with the CES‑D has reasonable power to detect a serious and persisting health problem that affects education and efficiency in the workforce.
Education often fulfills two roles: providing additional skills to a person and also providing the skills needed to acquire additional skills. On the job training is an important way of acquiring skills and this can come either in the form of formal training programs or in the form of learning on the job. There is a series of questions in the 1993 NLSY79 that deal with on-the-job learning, including how long it took the respondent to feel comfortable at his/her job.
An alternative to these is to consider some of the questions appearing in the core Reflex instrument, which alluded to above in A and B, which has been widely administered as a paper and pencil self-interview across Europe and elsewhere. I have attached that instrument. Questions that could be included in a CATI interview (assuming they have not already been asked) are: A6, A7, A9, C3, C4, C5, D1–4, D7–13, E1–7, F1–15, G13–21.
The questions in H1 and H2 are quite different from what we see in the US. They are intriguing questions, but before putting them in the NELS, a pilot data collection effort might be the place to start. For example, making contact with 5–10 diverse universities interested in follow-up with their graduates might lead to such a pilot whose results could be analyzed to assess the utility of moving in this direction in the future. A collaborative effort with the NSF baccalaureate survey might be mutually beneficial as a way of assessing some of the more qualitative aspects of education.
If these questions are appealing and the Department of Education is interested is comparative analysis between the US and the rest of the world using these questions, I can provide a lead.
3-
| File Type | application/msword |
| File Title | Appendix 3 |
| Author | cannada |
| Last Modified By | #Administrator |
| File Modified | 2011-02-15 |
| File Created | 2011-02-15 |