Instrument 19: Phone-based Skills Assessment Pilot Instrument

S upporting Statement for OMB Clearance Request

Instrument 19: HPOG 2.0 National Evaluation Phone-based Skills Assessment Pilot Instrument

National and Tribal Evaluation of the 2^nd Generation of the Health Profession Opportunity Grants (HPOG)

0970-0462

April 2019

Revised July 2019

Submitted by:

Office of Planning,
Research & Evaluation

Administration for Children & Families

U.S. Department of Health
and Human Services

Federal Project Officers:

Nicole Constance, Hilary Bruck, and Amelia Popham

Reviewer Notes:

Throughout Section B, the correct answers are printed in bold red type.

Introduction

Hello, my name is [ ]. May I please speak with _____?

IF RESPONDENT COMES TO THE PHONE: Hi, my name is [ ] and I work for Abt Associates, Inc., or Abt, an independent research company. Abt is helping the Administration for Children and Families in the U.S. Department of Health and Human Services with its evaluation of the Health Profession Opportunity Grants, or HPOG Program. We are conducting interviews with people like you who agreed to be in a study about a program offered at [NAME OF ORGANIZATION] called [PROGRAM NAME]. You joined the program in [MONTH AND YEAR OF RAD], when you applied to receive HPOG services. Thank you for taking the time to talk with me today.

We are getting ready to contact some of the people who joined the study so we can do a follow-up survey with them. Before we do that, we want to make sure that the survey instrument is working the way we want it to. By participating in this interview today, you are helping us test parts of the survey we will use with study participants. This test survey should take about 45 minutes to complete. I will ask you some questions about your educational activities, some basic questions about your use of computers, and some multiple choice questions on a variety of topics. Some of these questions may require you to determine the answer while we are on the phone. If you need me to repeat a question please let me know. You will receive a $25 gift certificate to a store of your choice, to thank you after you complete the survey.

Before we begin the survey, I would like to assure you that all of your responses on this survey will be kept private; your name will not appear in any written reports we produce. Your responses to these questions are completely voluntary. That means you may choose not to answer any question, or you may stop the interview if you wish, but we hope you don’t. Your responses to these questions will in no way affect your participation in any programs or your receipt of any kind of public benefits or services. The information you provide will be kept private and only used for this study.

According to the Paperwork Reduction Act (PRA), an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0462 and OMB approval expires on xx/xx/xxxx. If you have comments regarding this estimated interview length or any other aspect of this collection of information, including suggestions for reducing the interview length, please send them to Gretchen Locke Abt Associates, Inc. 10 Fawcett St. Cambridge, MA 02138; Attn: OMB-PRA 0970-0462. Do you have any questions before we begin?

[INTERVIEWER NOTE: RESPOND TO QUESTIONS AND THEN CONTINUE]

Let’s begin now.

1. Education and Computer Usage

Note to reviewers: questions in Section A are relevant for validating the focal questions for the pilot; questions in Section B are the focal questions for the pilot, which are being considered for Section J of the Intermediate Follow-up Survey; Section C questions are to clarify contact information for incentive purposes; and Section D contains questions related to the pilot feasibility.

The first set of questions asks about your current education.

1. What is the highest degree or level of school that you have completed? (SOURCE: VARIOUS CAREER PATHWAYS SURVEYS)

1. Did not complete high school

2. Regular high school diploma, GED, or other high school equivalency

3. Some college, but no diploma, certificate or degree

4. A diploma or certificate from a college requiring less than a year’s worth of credits

5. A diploma or certificate requiring a full year or more’s worth of credit (but less than an Associate’s degree)

6. Associate’s degree

7. Bachelor’s degree or higher (for example, BA or BS)

97. REFUSED

98. DON’T KNOW

2. People can get occupational training at different places besides a college. They can go to a local high school or technical school, a training institute, or they can get training through their employer. Have you received a certificate for completing a training course from someplace that is not a college? (SOURCE: HPOG 2.0 SHORT TERM SURVEY, MODIFIED)

1. YES

2. NO

7. REFUSED

8. DON’T KNOW

3. To work in some occupations, you may need a special license or certification. These special licenses or certifications often require you to pass a test to prove that you have mastered the skills for the occupation. Have you ever received such a license or certification? (SOURCE: HPOG 2.0 SHORT TERM SURVEY, MODIFIED)

1. YES

2. NO

7. REFUSED

8. DON’T KNOW

4. When did you last participate in any educational or training program of any type, including high school, postsecondary schools, and apprenticeships? Please give me the month and year you last participated. (SOURCE: HPOG 2.0 SHORT TERM SURVEY, MODIFIED)

___ ___ (1-12) / ___ ___ ___ ___ (2011-2020)

M     M /    Y    Y    Y    Y

5. IF VOLUNTEERED: NO TRAINING SINCE LEAVING HIGH SCHOOL

6. IF VOLUNTEERED: CURRENTLY ENROLLED

7. REFUSED

8. DON’T KNOW

Now I have a few questions about your use of computers.

5. In everyday life, how often do you usually...(SOURCE: PIACC)

	Never	Less than once a month	Less than once a week, but at least once a month	At least once a week, but not every day	Every day
Use email?					
Use the internet in order to better understand issues related to, for example, your health or illnesses, financial matters, or environmental issues?					
Conduct transactions on the internet, for example buying or selling products or services, or banking?					
Use spreadsheet software, for example Excel?					
Use a word processor, for example Word?					

B. Vocabulary and Arithmetic Assessment

Now I’m going to ask you a set of multiple choice questions on a variety of topics. Some of these questions may require you to calculate the answer before telling me your response. Please do not use a calculator. If you need me to repeat any question please let me know. I will be able to repeat each question once. We understand that some of these questions may seem hard. It is okay if you aren’t sure about an answer, we can just move on to the next one. As I mentioned at the start of our call, we are trying to see how well the questions work. Your input will help us make sure that these questions work the way we expect them to.

[NOTE TO REVIEWERS: correct responses to each question are in red font.]

[NOTE TO INTERVIEWERS: READ EACH QUESTION SLOWLY, PAUSE BETWEEN SENTENCES, PARTICULARLY IN NUMERACY QUESTIONS WHERE EACH SENTENCE IS ON A SEPARATE LINE.]

1. Your doctor has given you a new prescription with the following instructions: Take 2 pills twice daily with food for 7 days. Which of these choices represents the BEST way for you to take this medicine? (SOURCE: HL, #4)

1. Take 2 pills when you wake up and 2 before bed

2. Take 2 pills daily for 7 days

3. Take 2 pills with breakfast and 2 with dinner for 7 days.

7. REFUSED

8. DON’T KNOW

2. Your doctor offers you two drugs.

Drug A has a 1 in 100 chance of death. Drug B has a 99 in 100 chance of survival.

Which one is more risky? (SOURCE: HL, #9)

1. Drug A

2. Drug B

3. They are equally risky

7. REFUSED

8. DON’T KNOW

3. Which of these percentages represents the biggest risk of getting a disease? (SOURCE:FL,#1)

1. 1%

2. 10%

3. 5%

7. REFUSED

8. DON’T KNOW

4. The chance of getting a disease is 10%.

How many people out of 1,000 would be expected to get the disease? (SOURCE:FL, #5)

1. 100

2. 10

3. 90

4. 900

7. REFUSED

8. DON’T KNOW

The next set of questions asks about some common health-related issues.

5. If you have a rash or a brown spot on your skin, that your primary care doctor could not treat, what type of specialist would your doctor suggest you see to have it looked at?

(SOURCE:NEW)

1. a dermatologist

2. a cardiologist

3. an ophthalmologist

7. REFUSED

8. DON’T KNOW

6. Coronary artery disease can increase your risk for a heart attack. Which of the following things can cause coronary artery disease? (SOURCE:NEW)

1. obesity

2. smoking

3. high cholesterol

4. All of the above

7. REFUSED

8. DON’T KNOW

[NOTE TO INTERVIEWERS: READ SLOWLY AND PAUSE BETWEEN SENTENCES]

7. Lactose intolerance means that the body cannot digest foods with lactose in them.

Lactose is the sugar found in milk and foods made with milk.

After eating these foods, some people may feel sick to their stomach.

They may also have gas or diarrhea. Which of the following is a symptom of lactose intolerance? (SOURCE: HLSI-S1, modified)

1. Constipation

2. Stomach ache

3. Heartburn

7. REFUSED

8. DON’T KNOW

[NOTE TO INTERVIEWERS: READ SLOWLYAND PAUSE BETWEEN SENTENCES]

8. Five signs of a stroke are sudden changes that a person experiences:

(1) numbness or weakness of face or arm, especially on one side of the body;

(2) confusion, trouble speaking or understanding;

(3) trouble seeing in one or both eyes;

(4) trouble walking, dizziness, or lack of balance; and

(5) severe headache.

Which of the following is NOT a sign of stroke? (SOURCE: HLSI-S2)

1. Shaking or chills

2. Blurred vision

3. Bad headache

4. Numbness on one side

7. REFUSED

8. DON’T KNOW

9. Which of the following means that your body temperature is dangerously low? (SOURCE: NEW)

1. Hypoglycemia

2. Hypothermia

3. Hypochondria

7. REFUSED

8. DON’T KNOW

10. A prosthesis might be (SOURCE: NEW)

1. a mass in the abdomen

2. an artificial leg

3. something to read

7. REFUSED

8. DON’T KNOW

11. If you had an elevated risk of mortality, you would…(SOURCE: NEW)

1. have a higher chance of dying than most people

2. be likely to behave in an highly ethical way

3. have a chemical imbalance in the brain

7. REFUSED

8. DON’T KNOW

12. A deduction is defined as….(SOURCE: NEW)

1. Money taken out of a paycheck

2. A logical conclusion

3. Either a or b

7. REFUSED

8. DON’T KNOW

13. When would you give someone the Heimlich maneuver (also known as abdominal thrust)? When someone is: (SOURCE: NEW)

1. Sneezing

2. Choking

3. Coughing

7. REFUSED

8. DON’T KNOW

14. If you had a biopsy done, who would read the results? A… (SOURCE: NEW)

1. Podiatrist

2. Pathologist

3. Radiologist

7. REFUSED

8. DON’T KNOW

15. Which of the following statements is not true? (SOURCE: NEW)

1. Some doctors are physicians.

2. If you are a physician, you are also called “Doctor”

3. Everyone who has the title “doctor” is a physician

7. REFUSED

8. DON’T KNOW

16. Jasper is a male border collie who weighs 48 pounds. The average weight range for a male border collie dog is 35 to 45 pounds.

Is Jasper’s weight within the average range? (SOURCE: NEW)

1. Yes

2. No

7. REFUSED

8. DON’T KNOW

[NOTE TO INTERVIEWERS: READ SLOWLYAND PAUSE BETWEEN SENTENCES]

17. A patient drinks:

220 milliliters of soup,

100 milliliters of coffee, and

90 milliliters of juice for lunch.

How much liquid did that patient drink for lunch?

INTERVIEWER: RECORD RESPONSE VERBATIM (SOURCE: NEW)

REVIEWERS: CORRECT RESPONSE IS 410 mL

__________milliliters

7. REFUSED

8. DON’T KNOW

18. People with a cognitive impairment are likely to have difficulty… (SOURCE: NEW)

1. Speaking out loud.

2. Remembering and understanding conversations.

3. Hearing soft sounds.

7. REFUSED

8. DON’T KNOW

19. A phlebotomist is someone who … (SOURCE: NEW)

1. Operates on the front of the brain.

2. Draws blood from patients.

3. Classifies different kinds of diseases.

7. REFUSED

8. DON’T KNOW

20. Which of the following is an example of conversion to the metric system? (SOURCE: NEW)

1. Converting AC current to DC current.

2. Weighing something in pounds and recording the weight in kilograms.

3. Measuring a liquid in liters and reporting it in cups.

7. REFUSED

8. DON’T KNOW

[NOTE TO INTERVIEWERS: READ SLOWLYAND PAUSE BETWEEN SENTENCES]

21. Half of all runners that finish a marathon in less than 3 hours are expected to have knee problems.

Of those expected to have knee problems, half of them will require physical therapy.

There are 100 people running a marathon and half are expected to finish in less than 3 hours.

Of those who finish in less than three hours, how many runners are expected to require physical therapy?

INTERVIEWER: RECORD RESPONSE VERBATIM (SOURCE: NEW)

REVIEWERS: CORRECT RESONSE IS 25

__________________

7. REFUSED

8. DON’T KNOW

22. Mike picks Joe up for work and drives him home again 4 days each week. Over 5 weeks, how many days will Mike pick Joe up for work?

INTERVIEWER: RECORD RESPONSE VERBATIM (SOURCE: NEW)

REVIEWERS: CORRECT RESONSE IS 20

__________________

7. REFUSED

8. DON’T KNOW

23. Bilateral means….(SOURCE: NEW)

1. Having two sides

2. Coming to an agreement

3. Overcoming a disease

7. REFUSED

8. DON’T KNOW

24. Patients in a critical condition are expected to…. (SOURCE: NEW)

1. Complain a lot.

2. Be very near death.

3. Have different kinds of problems.

7. REFUSED

8. DON’T KNOW

25. Temperature can be measured in degrees using what scale? (SOURCE: NEW)

1. Fahrenheit.

2. Celsius.

3. Either a. or b.

4. Neither a. nor b.

7. REFUSED

8. DON’T KNOW

26. If you were taking a medical history you would be…. (SOURCE: NEW)

1. Asking questions to a patient.

2. Asking a doctor where they went to school.

3. Checking on Google or Medscape.

7. REFUSED

8. DON’T KNOW

27. If something is nocturnal, it would… (SOURCE: NEW)

1. Sing loudly.

2. Happen at night.

3. Wet the bed.

7. REFUSED

8. DON’T KNOW

28. Which of the following objects has bilateral symmetry? (SOURCE: NEW)

1. The human body.

2. A pair of gloves.

3. A snail shell.

7. REFUSED

8. DON’T KNOW

29. A median is (SOURCE: NEW)

1. The midpoint of a series of numbers.

2. A line down the middle of a body.

3. Either a. or b.

4. Neither a. nor b.

7. REFUSED

8. DON’T KNOW

30. What capital letter is composed of two vertical lines and one diagonal line? (SOURCE: NEW)

1. The letter Z.

2. The letter N.

3. The letter H.

7. REFUSED

8. DON’T KNOW

31. Which of the following is NOT an example of rank ordering something? (SOURCE: NEW)

1. Lining up children from the shortest to the tallest.

2. Arranging coins from the most valuable to the least valuable.

3. Grouping people by their eye color.

7. REFUSED

8. DON’T KNOW

32. Which of the following ratios are equivalent? (SOURCE: NEW)

1. One half and five tenths.

2. One third and two fifths

3. Seven tenths and four fifths

7. REFUSED

8. DON’T KNOW

33. If the drip rate from an IV bag is 2 milliliters per minute, how soon will it be necessary to replace a 320 milliliter bag? (SOURCE: NEW)

1. 160 minutes

2. 640 minutes

3. 32 minutes

7. REFUSED

8. DON’T KNOW

34. If a patient had a health insurance policy with large co-payments, that would mean…(SOURCE: NEW)

1. They might have a lot to pay up front at a doctor visit

2. They could get care without paying very much

3. They would have to show identification whenever they visited the doctor

7. REFUSED

8. DON’T KNOW

35. When doctors take a pledge called the Hippocratic Oath, what do they promise to do? (SOURCE: NEW)

1. Make sure that they do not harm patients

2. Never give different advice to different patients.

3. Always treat nurses with respect

7. REFUSED

8. DON’T KNOW

36. Out of every 100 African American men who die, 5 die of prostate cancer and 95 die from other causes. For every 100 White men, 3 die of prostate cancer and 97 die from other causes.

Based on the information I just gave you, who is more likely to die of prostate cancer? (SOURCE: HLSI-S4)

1. African American men

2. White men

7. REFUSED

8. DON’T KNOW

[NOTE TO INTERVIEWERS: READ SLOWLYAND PAUSE BETWEEN SENTENCES]

37. At Company A, 500 people have their annual performance review in June.

Half of the people being reviewed are expected to receive a raise.

Of those who receive a raise, one in ten is expected to also be promoted.

How many people who receive a raise are expected to be promoted? INTERVIEWER: RECORD RESPONSE VERBATIM (SOURCE: NEW)

REVIEWERS: CORRECT RESPONSE IS 25

__________________

7. REFUSED

8. DON’T KNOW

38. Jack and Jenny opened a small store called J and J’s Cookies last year. The company has been successful and it is growing quickly.

1. J and J’s Cookies had 50 employees last year. This year the number of employees working at J and J’s cookies doubled. How many people work at J and J’s cookies now?

INTERVIEWER: RECORD RESPONSE VERBATIM (SOURCE: NEW)

REVIEWERS: CORRECT RESPONSE IS 100

__________________

7. REFUSED

8. DON’T KNOW

39. People working in which of the following occupations do the same type of work?

(SOURCE: NEW)

1. A reporter and a swimming instructor

2. a teacher and a professor

3. a librarian and a doctor

7. REFUSED

8. DON’T KNOW

40. If you wanted to live in a state with a tropical climate, should you move to Alaska or Hawaii? (SOURCE: NEW)

1. Alaska

2. Hawaii

3. Either a. or b.

7. REFUSED

8. DON’T KNOW

41. You have a choice to enter one of two raffles. In Raffle A, you have a 10 percent chance of winning. In raffle B, you have a 1 percent chance of winning. Which raffle gives you a better chance of winning? (SOURCE: NEW)

1. Raffle B

2. Raffle A

7. REFUSED

8. DON’T KNOW

42. Mike will go to soccer practice three times a week. If Mike has soccer practice for 8 weeks, how many times will he go to soccer practice?

INTERVIEWER: RECORD RESPONSE VERBATIM (SOURCE: NEW)

REVIEWERS: CORRECT RESPONSE IS 24

1. :__________________

7. REFUSED

8. DON’T KNOW

43. A stop sign is an octagon. How many sides does a stop sign have? (SOURCE: NEW)

1. 2

2. 8

3. 6

7. REFUSED

8. DON’T KNOW

44. Mrs. Hooper’s students got the following scores on their spelling test—Maria got an 80, Juan got a 95, and Sharon got a 75. (SOURCE: NEW)

If you were to rank order the class with the highest score first, which is correct:

1. Juan, Maria, Sharon

2. Juan, Sharon, Maria

3. Maria, Sharon, Juan

7. REFUSED

8. DON’T KNOW

45. If the vet tells you that your puppy is hyper-active, does that mean that he is (SOURCE: NEW)

1. Overly active

2. Not very active

3. Neither a. nor b.

7. REFUSED

8. DON’T KNOW

C. Closing

Before we complete this portion of the interview, I would also like to make sure I have your contact information recorded correctly. We will use this information to email you a link to a website where you can redeem your $25 gift card.

1. I have your name recorded as [FIRST MI LAST]. Is this still correct or have you changed your name?

1. YES, STILL CORRECT [SKIP TO C2]

2. NO, NAME CHANGED

1. What is your first name now?

2. What is your middle initial now?

3. What is your last name now?

2. As I mentioned at the start of this interview, we will send you an email with a link for you to login to a website and select your $25 gift card to one of the available stores. In order to make sure you can receive your gift card, I need to be sure I have your email address.

   1. What is the best email address for us to reach you at?

____________________________@____________.______

6. IF VOLUNTEERED: I DO NOT HAVE AN EMAIL ADDRESS

Read: Okay thank you for letting me know that. We can make arrangements to mail your gift card to you.

3. I want to make sure I have your correct phone number in case we need to reach you for any reason. I have your primary phone number recorded as [xxx-xxx-xxxx]. Is this still correct or do you have a new primary phone number?

1. YES, STILL CORRECT [SKIP TO CLOSING]

2. NO, CHANGED

1. What is the new number, starting with the area code?

___ ___ ___ - ___ ___ ___ - ___ ___ ___ ___

2. Is that a home, cell, work, or other number?

1. Home

2. Cell

3. Work

4. Other

CLOSING: Thank you very much for your time today.

IF RESPONDENT HAS A VALID EMAIL ADDRESS: You should receive an email with a unique link to redeem your $25 gift card within the next four weeks.

IF RESPONDENT DOES NOT HAVE A VALID EMAIL ADDRESS: You should receive a $25 gift card in the mail within the next four weeks.

D. Interviewer Feedback Questions

INTERVIEWER: COMPLETE AFTER YOU FINISH THE INTERVIEW WITH RESPONDENT

1. Did the participant have any reaction when you transitioned from the survey items in Section A to the assessment items in Section B?

1. YES

2. NO (SKIP TO D2

8. DON’T RECALL

D1a. If yes, what were the respondent’s reactions?

2. How easy was this instrument to administer?

   1. Very easy to administer (SKIP TO D3)

   2. Somewhat easy to administer (SKIP TO D3)

   3. Hard to administer

D2a. Why was it hard to administer?

3. Did the participant ask for any questions to be repeated?

1. YES

2. NO (SKIP TO D4)

8. DON’T RECALL

If yes, which questions did the participant want repeated?

Question #____________________________

4. Were there any questions that caused the respondent to pause for a long period of time before responding?

   1. YES

   2. NO (SKIP TO D5)

8. DON’T RECALL

If yes, which questions caused the participant to pause?

Question #____________________________

5. Did the respondent start to show signs of interview fatigue? If so, at what point did that happen?

[INTERVIEWER: signs of fatigue may include change in the respondent’s tone of voice, frequent pauses or sighs, increasing requests to repeat a question or response option, or they clearly divert their attention to something else going on around them.]

1. YES

2. NO (SKIP TO D6)

8. DON’T RECALL

If yes, at what point did that happen?

Question #____________________________

6. Is there anything else you would like to note about the interview?