Preliminary Findings Report

Forms B, C, and D: Cognitive Testing of
HSLS Field Test Items

Preliminary Findings

Research Support Services, Inc.

January 7, 2011

This brief report summarizes findings of the cognitive testing conducted by Research Support Services for HSLS study. The cognitive interviews were conducted in Illinois, in the Chicago SMSA from December 27, 2010 through January 6, 2011. Eight parent interviews were conducted with Form B. Six student interviews were conducted with Form C and six with Form D.

The following table displays respondent characteristics.

Parent Interviews Student Interviews

R#	Gender	Ethnicity
AS001	F	White
AS002	F	White
AS003	F	African-American
LA001	F	White
KK001	M	White
KK002	F	African-American
KK003	F	White
KK004	M	White
KK007	F	Mixed

R#

Gender

Race-Ethnicity

Form Tested

AS001

F

White

C

AS002

F

White

C

AS003

F

African-American

C

AS004

F

African-American

C

LA001

M

White

D

KK001

F

White

C

KK002

M

African American

C

KK003

F

White

D

KK004

F

White

D

KK005

M

White

D

KK006

M

African-American

D

KK007

F

African-American

D

Recruiting was done by word of mouth and by having a parent distribute flyers among other families of juniors. Participating students attend four different high schools, including a parochial school.

Findings from Testing Form B

1. During the last 12 months, which of the following activities have you or another family member done with your teenager?

Parents able to think both about other family members and self. A parent thought only from last summer through present.

Parents wondered what would go under ‘Other’. Educational activities done by the family? How about sports? What about art museums?

Parents covered wide range of things under the answers: Answered ‘build or fixed something’ thinking of when she asked son to find part for her dishwasher on internet. Answered ‘worked or played on a computer together’ but they never sit side by side. Answered ‘gone to a play, concert, or live show’ thinking only of school performances that involved her son. Another parent checked ‘discussed a program or article about math, science or technology’ because the student worked on a physics project for which she had to comply with the IIT competition rules and they discussed that.

A parent indicated that visiting a library and attending a school fair are things she did with her child in earlier years but not in high school, and that there are not science fairs in the high school.

KK001/002/003/007: R stated had done additional activities when student was younger.

KK002: Added other specify ‘sporting events’

KK003: R marked ‘worked on computer’, in probe R stated email all the time or student showing R you tube video. R marked discussed a article, in probe stated it was homework only that was discussed.

KK004: R noted that computers in general are not used as much as they were ‘five years ago’, that today’s cell phones are technology in our hands. R did not originally mark a museum although they visited a museum while on vacation this summer. R saw museum in the question as a ‘big, bulky thing, old structures’, the museum that they visited was a small local museum. R noted he would also not have included petting zoos, ‘holiday zoos with donkey’s or farm zoos.

KK007: In probe, R noted that they had done ‘culturally significant activities’ this year, such as visiting Mayan Ruins “where we learned about their phenotype and genotype and things of that nature” but that it didn’t fit in any of the categories (museum like but not a museum). R had not marked library; they visited this summer to do research for summer school projects but read the question as ‘visited’ as in for pleasure, not for ‘academia’. Also, has not gone to live shows but often watches on TV or video with her.

2. By the time your teenager graduates from high school, do you think he/she will have met the minimum requirements needed for admission to….

All but one parent answered yes to both. Parents based their answer on the high school making sure through the counselors that students take the courses they need to meet the minimum requirements. ‘Minimum requirements’ was well understood.

Technical Institute = associated with computer training, DeVry Institute AS001

AS002: ITT / DeVry computer training something like that

KK004: Parent marked only 4 year, stating that it was obvious that if she could meet requirements for a four year she could meet them for a two year.

LA001: Technical institute: a certified program that would give you some sort of computer certification or maybe a mechanics certification or child care certification or nursing

3. How have you gotten information about financial aid?

AS001/002: Not gotten any. No problem.

KK001: R responded ‘researched on internet’. R has done all activities listed with older sons in the last two years but stated he unsure if he should mark them, since he already knew about them.

KK002: R marked only ‘researched on internet’ but stated in later questions that she had gone through the whole financial aid process with her older child who has scholarships, loans and work study.

KK003: R added to ‘Other’ ‘Talked with other parents’

KK004: R responded ‘researched on the internet’. States he was thinking of this student specifically, he has used all methods with two older sons. Also noted that he used a network of friends and family for information in the past.

LA001: R marked ‘talked to a school counselor’ but indicated she chatted for 2 minutes.

4. Will you apply for financial aid for your teenager’s education after high school?

No issues generally. One parent marked DK because student was offered an athletic scholarship but won’t know how much $$ until senior year.

4A. What types of financial aid do you think your teenager would qualify for?

Generally, no issues.

LA001: Parent wondered if small awards through the HS would count as financial aid

4B. Why will you not apply for financial aid?

Most parents were not asked this due to skip pattern.

AS001: Other: athletic scholarship

5. Families have different ideas about what they would like their teenagers to do in the first year after high school. How strongly would you encourage or discourage your teenager from choosing each of the following activities as his/her main activity?

Parents kept clearly in mind that the question referred to the first year after high school and was not about long term choices. For enrolling in further education they spoke about the importance to continue studying and that taking a break sometimes makes it harder to come back. They appeared able to distinguish between their likes/dislikes and whether they would encourage/discourage.

Parents distinguished the strength of discourage across categories. They expressed stronger discouragement for some things than for others marked with the same rating.

KK001: R answered discourage for both armed services and volunteer work; if given a broader range, would have put armed services slightly higher than volunteer work.

KK002: R responded ‘encourage’ for missionary/volunteer, R stated originally read as part time, but would keep encourage if full time as long he continued to go to school full time.

KK003: R wrote in ‘get a job’ because she saw this as a common option. R wrote encourage for volunteer or missionary work. Stated that it was not exclusive and she had seen it as either full time or part time. She’d neither encourage nor discourage missionary work if she wanted to take a year off because her teenager would have to finance it herself and wouldn’t be able to do so.

KK004: R put neither encourage nor discourage for all but education (strongly encourage). Stated that he couldn’t take a position on these since they’d never been considered.

KK007: R had marked neither encourage nor discourage volunteer/missionary, she saw this activity as in addition to schooling (strongly encourage), if a separate ‘main activity’ she would strongly discourage unless was time between high school and college.

LA001: no issues

6. If there were no barriers, how far in school would you want your teenager to go?

Parents included under barriers: Financial, motivation, balance with family time. Generally no issues.

KK001: R not clear about level which student needs to reach to fulfill career goals.

6A. As things stand now, how far in school do you think your teenager will actually get?

No issues in this question.

KK002: Found ‘you don’t care’ as very odd

6B. Students often have circumstances which make it difficult for them to complete the level of education that their parents hope they will. Would you say that your teenager may not complete the degree or diploma you hope he/she will get because…

Generally, no issues in this question.

KK002: R marked “other’ ‘lack of commitment. Did not mark ‘grades are not high enough’ or ‘taken the right courses’ because she said she had no idea how well he would do in college or what courses would be required for whatever he chose to do. R also clarified while answering the question, she thought solely about completing bachelors’ degree, not moving from bachelors to PhD (goal education).

7. How important to you would each of the following colleges’ characteristics be when choosing a college or university for your teenager to attend?

One parent felt a category was missing: providing a good education

Academic reputation: parents defined in different ways, placing grads in jobs, pretentiousness, how well it is known, assumption of being ‘smart’ if you are a graduate; accredited. Not interpreted as academic excellence.

Campus safety: parents noted both security procedures on campus (locks, escort services) as well as location of campus perceived dangerous areas (urban)

Campus appearance- Parents included both the grounds of the campus and the facilities and dorms

Location: Parents defined either in terms of distance to them or in terms of location within urban/rural

A good record of placing graduates in jobs: Parents mentioned reputation as well as internships, job fairs

A good record of placing graduates in graduate or professional schools: Parents saw this as preparatory and mentioned reputation

Recommended by family or friends: Parents include both their own conversations and conversations their students have had with family members, such as older cousins.

8. Approximately what percentage of the total costs of his/her education do you think he/she should finance?

Most problematic of all questions. Parents generally answered it depends on how much the cost of the education ends up being. If the student goes to a school where the total cost of tuition/room/board is $50,000, the percentage the student would be expected to pay would be very different from that if the student gets scholarships and they have to pay $10,000. Also, differences depending on which school they go to (e.g. Harvard vs. state school).

AS001: 10%?

AS002: “Never really thought about it this way. Maybe 20%?”

KK001: >50% Parent believes the student will be vested in the education if she has to pay more than half

KK002: 40%

KK003: 50%, includes loans

KK004: 40%

KK007: 0

LA001: as much as able then summer part time employment – could not answer – percentage can be impossible if school is costly

9. Which of the following steps do you expect him/her to take to finance his/her education?

Parents with college intentions and not many worries about funding education answered the question thinking of what student already does, but not always thinking of the student doing them to actually finance their education. More for spending money than ‘to finance education’.

KK001: Includes student loans as funding.

KK004: R includes student loans and all resources that the student activity works to get, including scholarships that she completes the application for or actively finds in internet searches.

Findings from Testing Form C

1. Do you plan to go to college after high school?

Students had no significant problems with this question.

‘They were not familiar with ‘career college’ but guessed that it was a college level school that trained you for a specific career, another student guessed it’s like a vocational school. They were not familiar with ‘technical institute’ but several guessed it was a computer training school, ‘such as DeVry or ITT’ (both of these advertise widely in the media in the Chicago area and are probably the only instances of respondents ever hearing about ‘technical institutes’).

AS003: “Is technical institute like a vocational school?”

AS004: Marked 4-year college. In probing she explained she wants to go to a 2-year college and then transfer to a 4-year college. If intention is about where they want to go immediately after HS, then recommendation is to say that specifically.

“Career college” -> training for a specific job (AS004) - “like in an internship or on the job”.

KK001: “Isn’t [a technical institute] like computers, computer technology, things like that?” “I don’t even know what a career college is, like a nursing college maybe?”

KK002: Technical institute – ‘It’s a college where you learn a trade, like automotive or culinary arts’, community college- somewhere you go if you don’t have enough money you can take your freshman and sophomore year for a four year institute. Career college- a specific field like ITT tech.

2. Which of the following reasons apply to your decision to go to college? Mark all that apply, then pick the strongest reason.

Your parents expect you to

You don’t know what else you would do

Your friends are doing the same thing

You are interested in a certain subject or field

You like school

Other, please specify

There were no problems in marking multiple choices and picking the strongest. One R ranked her top two choices and said they are fairly equally strong.

AS001 picked strongest – interested in a subject (also marked 1 and 3) – reason that has most influence on you – interpreted question as asking about outside influences – Might add a response about ‘for your own future’.

AS002 want to do something with my life, for myself – (R said no choice offered covered that)

AS004: strongest was Other (Specified: “I understand that my future will not lead me onto the path that I want, without me deciding to.”) Q to her asks: “Why do I want to go to college”

KK001: Hand wrote in ‘support self’. R marked 1, 4 &6, did not pick strongest.

KK002: R picked 1 & 4, did not pick strongest. My parents want me to because you can’t work at McDonalds or Jewel anymore and support yourself. R wants a degree in marketing because of a computer graphics class where they worked on logos that was really interesting.

3. Which of the following qualities are you looking for in a college?

Academic reputation

AS004: want to be able to brag about it

KK001: Its only people who are really intelligent, good GPAs, ACT scores, they will be the only ones who get into the good colleges

KK002: The better the school the more opportunity you have to be successful

Campus safety

AS002: wants security in school

AS003: concern because of a recent case

KK002: concern from recent case where girl ‘came up missing’

Financial aid

AS002: wants to avoid starting a job after college and be in debt

Campus appearance

AS002: not in middle of nowhere, nothing to do

AS004: cleanliness of campus dorms

Location

AS002: combined with campus appearance – wants places around to do things outside of campus

AS003: it has to be in a good neighborhood (tied to campus safety for this R) – not a big natural disaster area

AS004: want to branch out, get away

A good record of placing graduates in jobs

AS002: to know she will be able to get a good job after school

KK001: …they help them find jobs by having a education and different programs like internships

Opportunity to play sports

AS002: to be able to play softball

KK001: I thought of [varsity sports], I didn’t think of intramural or anything

KK002: I want to play sports (varsity level)

Recommended by family or friends

AS002: if you see where someone is in life and they are doing well, the college they went to must have done something right.

4. What field of study are you considering? Check all that apply.

Good understanding of field of study. Students not knowledgeable about what humanities are, and one not sure about social sciences. List comprehensive and non duplicative. One R thought Arts was not on the list, then saw Liberal Arts and corrected herself. One student associated ‘protective services’ with social work, but examples dissuaded her that it was not the same.

AS001: Education or teaching (wants to be HS teacher) -

AS002: Education or teaching. Not sure what humanities means.

AS003: Health professions (wants to be a doctor) / Education or teaching (would love to teach young children)

AS004: Health professions. (wants to be a doctor or psychologist)

KK001: Health Sciences (wants to be physical therapist) Not sure what Social Sciences and History. Not sure are, not sure what kind of job you can get from those. Liberal arts and humanities? Helping people? Like helping build houses, something like that.

KK002: Business. [probed what are physical sciences] – kind of like doctors and I would say people who do biological stuff [and how is this different than biology] “because I think physical science would be like if the company is making a product like Gatorade…you’d have to know how the product will react to the body.” Humanities-like politics; Social sciences- like a history teacher or archeologist.

5. Earlier, you told us you planned to enroll in college after you finished high school. Which career college, community college or university would you most likely attend?

Juniors at this stage appear to be answering about what school they would most like to attend, rather than about likelihood.

AS001: ISU – visited overnight – likes academics, distance from home, campus appearance, good fit

AS003: DK because she always wanted to go to top schools but now knows her grades are not good enough, so she needs to rethink.

AS004: Augustana College – Although R wrote this in, it appears to be her ultimate goal but not her immediate goal after HS. She plans to go to community college, save up, and eventually transfer

KK001: Northwestern – has a good physical therapy program

KK002: Illinois (UofI) – close, heard campus looked nice

6. Why do you have this specific college in mind? Please choose the most important reason:

No serious problems detected

AS001: academics is most important but reputation can be a false reputation. R chose opportunity to play sports because she has been offered scholarship there. Maybe academic quality would be better description.

AS004: other: “great sounding choir”

KK001: R did not choose a single response probed to Academic Reputation ACTs good schooling the professors are well educated. R also added location - both to home and to what’s around the campus, like things to eat, bowling. Campus appearance – first impression, dorms, how far it is to walk from one class to another. Good record of placing jobs- I’d be ready for the career world

KK002: R listed academic reputation, campus safety,& location. Business school is supposed to be good

7. How much does the following statement apply to you?

My family and I have talked about finding ways to pay for education after high school.

No serious problems detected

AS001: have talked SOME. Not a very big topic because it’s in state and offered scholarship

AS003: R spoke about thinking about it rather than TALKING about it. She is concerned that it’s not just her but also her little sister going to college that have to be considered.

AS004: R chose a little, but described as having talked VERY LITTLE

KK001: R chose a little. Described a little and some as ‘a little is just a few times, some is more than that, they are kind of close in meaning.’

KK002: R chose a little. States hasn’t really talked with parents about paying for college, just scholarships.

8. Difficulty in paying for college will keep me from attending school.

Respondents seemed to be answering about attending a specific school, not about attending school entirely.

AS001: a little – R has picked school but should she decided on another school, like Stanford, it would be very expensive and that might be a deterrent.

AS003: without the money, who can go? But I can start at a community college, that should jump start me.

AS004: A lot – R does not think parent will be able to help her

KK001: Some – R clarified that it wouldn’t keep her from going to college just maybe a ‘higher paid college’

KK002: A little – a little is like a side conversation but we haven’t really talked. “I put a little because if I can’t afford the one college, there is another I can attend.” A little and some mean the same thing because of the number of colleges out there.

9. On average, how much do you think a year of school costs at the college you listed in question 5? Please do not include living costs such as on-campus housing (room or board) or rent and utilities.

Respondents did not appear sure at all of their answers.

AS001: $25K – Seems to recall from campus visit.

AS002: $30K – heard

AS003: $50K – just tuition for the year - R&B will probably be $400

AS004: marked DK – upon probing said $15K

KK001: 30k – brother was at UofI and it cost 20k and he told her Northwestern was more so she guessed

KK002: DK – doesn’t know how to separate, thinks total costs at UofI are around $14,000

10. To help pay for their education, students sometimes receive scholarships or grants that do not have to be paid back. How much of this help do you think you would receive each year if you went to the college you listed in question 5?

No serious problems in interpretation, but Rs did not know what to answer.

AS001: R does not know because until Sr. year the University won’t make a firm offer of scholarship

AS002: R does not know. Depends on whether she gets a softball scholarship

AS003: $6000 because at the college fair one of the schools said they would give me that amount for my education.

AS004: DK but imagines perhaps her mom (parents separated) having a lower income than dad may mean she can get some scholarship.

KK001: DK hopes to get an athletic scholarship but won’t know if she’ll be recruited

KK002: DK – doesn’t know how much of athletic or academic scholarship he can get. Good understanding of loans, sees scholarships for athletics and grants for merit.

11. How much would you or your family be willing to borrow in the form of a loan to pay for a year of school at the college you listed in question 5?

Respondents did not appear to know this much.

AS001: DK – Partly depends on scholarship – Well understood

AS002: DK

AS003: $30,000 - chose that because she figured a little more than half. Based answer on how she thinks the family feels about borrowing.

AS004: “my parents will not accept help” – “They will probably be scared about paying it back.”

KK001: $10,000

KK002: $10,000 – sister got that much.

Findings from Testing Student Form D

1. Have you participated in any of the following activities to prepare for life after high school?

This presented assorted small interpretation issues:

KK003: R marked visited a college campus, during probe it was determined that she had been to a college only to watch a gymnastics meet and to visit a cousin but didn’t see it as looking at a college for herself.

KK004: R (high achieving student) unable to define ‘apprenticeship’. Only knew of it in terms of TV show The Apprentice, which she has heard of but never seen.

KK005: R stated confusion between categories job shadowing and internship since R sees it as same thing. R described conversations with teachers but stated that he did not check response category because the kind of college he wants to attend might be different. For ‘Other’ R did not mark anything but in probing noted that he met with an alumni group for a college he is interested in and has also met with the band representative of a college he is interested in where he would play in the band.

KK006: R marked yes for job shadowing, when probed stated that he wasn’t sure what it meant but thought it meant looking over what he wanted to do. In probing R has not had job shadowing experience.

KK007: R included both face to face and text messages as ‘talked’ in response ‘talked with friends’. Suggested adding ‘talked to someone about a career interest’ (outside of a job fair situation) when asked about additional ways to prepare for life after high school.

LA001: For R ‘career day’ and ‘college night’ sounded like the same thing. He also felt talking to teachers and talking to counselors was somewhat duplicative.

2. What do you look forward to in school?

Interpretation of school as HS vs school as College is the main issue here.

KK003: Marked ‘other’; wrote in ‘more than one thing equally’. In probing, R believed school to mean ‘college’ only.

KK004: Interpreted ‘school’ to mean college.

KK005: marked both seeing friends and art music and theatre and said he could not choose because they went hand in hand. Also noted when probed about Other, that he had considered adding ‘just learning in general, not a particular subject but just the idea of acquiring the stuff I need for later in life’

KK006: R entered Other specify=auto technology

KK007: No issues

LA001: R did not notice instruction and checked multiple answers. He indicated ‘history’ is missing from the list.

3. Have you taken any courses that have a specific occupational orientation? These include courses in areas such as agriculture, business, computer technologies, construction, or health care.

Most students had interpretation issues in this question regarding ‘courses with a specific occupational orientation’ despite the examples.

KK003: R was confused by the question. Not sure what courses this exactly referred to.

KK004: R marked yes; when probed, answered AP Science, AP Bio, AP Chemistry, when probed about specific occupational orientation, stated that she read it as what you plan on majoring in the future. When asked about each of the listed careers and what classes would fit, she stated school had agriculture=Environmental science class (AP); construction=welding & mechanics shop; health care= ‘probably just science’.

KK005: R interpreted to mean ‘the same kind of thing you would learn in college, one job, one occupation’.

KK006: No issues

KK007: R interpreted question to mean AP classes

LA001: R answered Yes. He took a summer course for HS students at a local community college, called “Careers in Engineering”. R interepreted ‘that have a specific occupational orientation’ as referring to courses that would have application in real life jobs.

4. What was the MAIN reason for taking these courses?

Question was appropriately skipped by most respondents. It did not present problems for the two who answered it.

KK006: R chose multiple responses, personal interest, prepare for job, explore career, learn in general

LA001: R answered To meet graduation requirements. The class fulfilled the Applied Art graduation requirement.

5. What is the MAIN reason you haven't taken such a course?

No problems in this question.

6. Have you received college credit for any of your high school courses?

This question presented problems. Students in the middle of Junior year do not yet know about college credit they will actually receive. Many are just taking AP courses now and have not even taken tests for them.

KK003: R stated yes and listed AP classes she has taken and is currently taken.

KK004: R stated question ‘confused her’ because she won’t find out if she receives credit (for her AP classes) until she enrolls in college. Also included local community college classes she intends to take next year on community college campus. Probed for other ways to get college credit and she stated several seniors she knows are taking online classes.

KK005: R interpreted question as college entry requirements not college credit, for example, he stated that he had taken one more math than what they require and will have taken two extra years of Spanish over the two required.

KK006: R was unsure if honors level as well as AP classes get college credit. He has taken neither.

KK007: R properly described how AP credit is given (as a high school course for high school and potentially a college credit once you start college at a college that accepts that credit). R maintained that AP classes are ‘for college credit’.

LA001: R wrote in ‘what if currently enrolled? Have not yet taken an AP exam’. Did not pick an answer.

7. Which one of the following BEST describes how you have decided which high school courses to take this year and next year?

Question did not specifically say Mark one only. Two Rs marked multiple answers.

KK004: (college-bound student) R responded courses to prepare for a job after high school; at probe R indicated he had interpreted it to mean after college.

KK005: R marked two responses, courses needed to meet and courses needed to do well. After probing was able to choose one and when asked about what ‘best describes means’ he stated that he answered that these two best describe why he decided.

KK007: R is honors level student, marked to meet high school requirements because she was not sure if the she had taken the exact classes needed to get into the college of her choice or which classes would help her do well in college.

LA001: R picked all answers except second choice. He indicated a missing choice is: ‘because it’s a challenging course that colleges would like to see on my transcript.’

8. How much does the following statement apply to you? Difficulty in paying for college will keep me from attending school.

Three Rs interpreted this as referring to a specific school, not to school entirely.

KK003: R responded a little. Stated it would not preclude her from a school, but she would not choose to go to a school that is ‘outrageous’.

KK004: R responded a little, interpreted to mean may not be able to go to most expensive college.

KK006: R responded some. Struggled with question. Stated it might keep him from attending a campus college, but he doesn’t want to go to a campus college. When asked if difficulty paying would keep him from a technical college (R wants to go into Automotive technology) he stated emphatically not at all.

9. How much money do you think you would earn in a year of working if you had the following levels of education? Please provide your best guess.

Respondents have very varied notions of this, and generally guessed with little information.

KK003: R wrote next to answers ‘I have no idea’, was unable to give a guess, emphatic that she couldn’t answer, that she has no idea what anyone makes and doesn’t care to and doubts that anyone could answer the question.

KK004: R stated she guessed and has a cousin with a GED/getting a GED who is currently making about $40,000 a year. Values were HS=35,000 (less than GED); GED=45,000; diploma=55,000 (higher than GED and value of her car at 9,000); College=75,000 (cousin who makes 75-80000 )

KK005: R is caddy in summer and has part time job, he used this knowledge to estimate left HS today=20,000. He believed GED was greater than HS so=50,000; Finish HS=30,000 and had done research on his career and had read starting salary with business degree =$60,000 for bachelors.

KK006: R stated he guessed. Estimates left high school=20,000; GED=25,000; diploma=35,000, finish college (technical college)=50,000. When asked about further education, i.e. Doctor, stated $100,000.

KK007: R stated she guessed. She estimated $2000 for left high school today; she was able to give reasonable estimates for electricity and phone costs per month but was unable to extrapolate overall costs to a year. She confirmed the $2000 was for a year. Other values were 4,000/GED, 7,000/HS, 19,000/college. She stated that 19,000 seemed right if earning about $60/hour seven days a week.

LA001: R asked: “The first year?” No other issues. Answered $20,000/$35,000/$40,000/$80,000

10. How certain are you about your answers to the previous question? Please answer from 0 to 100, with 0 being “completely uncertain” and 100 being “completely certain.”

No problems with using the scale.

KK003: R had certainty of 0

KK004: R had certainty of 60

KK005: R had certainty of 65

KK006: R had certainty of 95

KK007: R had certainty of 0

LA001: R had certainty of 25