Feedback on Current Personal Responsibility Education Program (PREP) Information Collections

Formative Data Collections for ACF Research

Instrument 1 Grant Recipient Focus Group Discussion Protocol.pptx

Feedback on Current Personal Responsibility Education Program (PREP) Information Collections

OMB: 0970-0356

Document [pptx]

Download: pptx | pdf

Revising PREP Performance Measures Participant Entry and Exit Surveys

September ##, 2024

[PREIS/Tribal/Competitive/State PREP] Listening Session

Lara Hulsey • Tressa Stapleton • Sydney Summers-Knight

Presented by Mathematica

Agenda

•Overview of PREP performance measures revision process
•PREP participant entry and exit surveys

- –Questions on participant characteristics
- –Questions on participant behaviors at entry
- –Questions on perceptions of program effects
- –Questions on program experiences
•Additional comments and next steps

Our agenda today will begin with a brief overview of the goals, process, and timeline for the revisions to the PREP performance measures. Then we’ll spend most of our time discussing the PREP participant entry and exit surveys, section by section, and how they can be improved. At the end there should be time for any other comments you might have, and we’ll conclude the webinar with a summary of the next steps.

Participation and Technical Assistance

•Participants can control mute and camera functions
•If you experience technical difficulties, request help through the chat

- –Select “Host” from the “To” drop-down list before sending your message

Everyone should be able to unmute yourself and turn on your camera. We hope that you will use these features, because participation is key for a listening session, where the goal is for our team to hear from you [as opposed to a training webinar, where the main purpose is for our team to provide information, and input is limited to the chat].

We encourage you to turn on your cameras, especially when you are speaking. (However, if you have any bandwidth issues arise when you turn on your camera, you might try turning it off to see if that improves the Webex quality.)

If you experience any technical difficulties during this listening session, please request help through the chat, selecting “Host” from the “To” drop-down list.

OMB Statement

•Please note that we estimate this will take about 90 minutes per respondent, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB is #0970-0356 and the expiration date is 01/31/2027. If you have any comments on this collection of information, please contact [email protected].

As a reminder, your participation in today’s conversation is voluntary, and will not affect your grant. You can decline to answer any questions you do not wish to answer. We are not recording the call, and nothing you say will be identified with your name in the memo we prepare for ACF summarizing this discussion. Although we can’t guarantee that no one else on the call will not share what we discuss today, I’d like to ask everyone to keep private the information that other participants share.

Revising PREP Performance Measures

Performance Measures Revisions
Goals and Process

Goals	Activities
Ensure the measures meet the data needs of ACF and grant recipients	Listening sessions with: oPREP grant recipients State PREP Competitive PREP Tribal PREP PREIS oFYSB project officers
Improve clarity of existing participant entry and exit survey questions
Incorporate more complete response options for some items to better represent the range of participants	Cognitive pre-testing of new items on sexual orientation and gender identity (SOGI)
	Adoption of new Office of Management and Budget (OMB) guidance on race/ethnicity items

ACF has several goals for undertaking revisions to the PREP performance measures participant entry and exit surveys. This slide summarizes those goals, along with the key activities planned to achieve them.

Starting with goals:

•First, to ensure that the performance measures meet the data needs of both ACF and grant recipients
•Second, we've heard from grantees that some youth respondents have a hard time understanding some of the questions in the entry and exit surveys or distinguishing between some response categories--which make it difficult for them to answer the questions. We would like to improve the clarity of the wording of existing questions, and to help ensure that youth respondents understand them (which in turn will improve the quality of the data)
•Third, both in response to comments from grantees and to align with broader federal initiatives, for certain questions on participant characteristics, we need to incorporate response options that better represent the range of youth participants. In particular:

- •Grantees have requested we revisit measures of sexual orientation and gender identity (SOGI) because the existing binary question does not include response categories that represent all youth. The government agrees, and the Office of Management and Budget (OMB) has required federal agencies to revise SOGI measures used in data collections.
- •We also recognize that youth have a hard time understanding ethnicity. The government has realized this as well, and OMB has issued guidance specifying new race categories that federal offices must use in data collection.

As we go through the revisions process, we’ll need to balance the ambitious goals of this effort with each other as well as the desire for consistency, which is important for assessing progress of PREP over time. We [might not/probably won’t] be able to implement every suggestion you provide today, but we will consider them all, and we greatly appreciate your input.

Also, note that our conversation today will focus on the performance measures collected through the participant entry and exit surveys. ACF is not currently planning to make any revisions to the Structure, Cost, and Support or Attendance, Reach, and Dosage measures at this time.

Turning to the second column, input into the survey revisions will come through three main avenues:

•First, we’re conducting listening sessions like this one with PREP grant recipients and with FYSB project officers. These listening sessions will cover the full set of participant entry and exit survey items,

and are intended to focus on the first two goals shown in the table

•The last/other two activities in the 2nd column on this slide each focus on a specific topic, and are part of broader changes within the federal government to improve questions agencies ask about particular demographic characteristics:

- •As part of an ACF-wide effort that is broader than just PREP, a team will be conducting cognitive pre-tests with youth of potential new items on sexual orientation and gender identity (SOGI). This effort is in part a response to a White House Executive Order 14075 requiring all government agencies to revise their questions on sexual orientation and gender identity. ACF’s testing [of recommended SOGI measures] will include PREP youth participants [as well as youth served by other ACF programs] and will occur in tandem with other activities to revise performance measures.

- - •In the cognitive pretesting, youth will first be asked to respond to several potential SOGI survey questions and will then be interviewed about those questions, to assess whether they understood them, so the team can take steps to address any problems.
- •Turning to the last bullet: based on feedback from communities, the U.S. Census Bureau recently updated the race questions they use and combined them with past questions on ethnicity, and [the Office of Management and Budget] OMB has developed new guidance on these race/ethnicity items, which we need to adopt. The new race categories are being implemented across government [and we’ll say a bit more about the categories on a later slide].

Expected Timeline for Revisions

•Input gathering – fall 2024
•Preliminary revisions – winter 2025
•OMB review – spring/summer 2025
•Introduce approved revisions – September 2025
•Begin administering revised surveys:

- oOptional – fall 2025
- oRequired for all PREP grant recipients – Jan. 1, 2026
•Submit data from revised surveys:
- oOptional – Jan./Feb. 2026
- oRequired for all PREP grant recipients – July/Aug. 2026

We are now at the very beginning of the process—the input gathering phase, with the activities discussed on the previous slide. After we’ve completed the listening sessions we’ll discuss your input with ACF and make recommendations for revisions to the measures. Once ACF approves a set of revisions, the revised measures will be submitted for OMB review. [We anticipate receiving OMB approval by September 2025. ] After OMB approval, we understand that grantees will need to submit the revised surveys to their IRBs or other local approvals.

Therefore, FYSB plans to give grantees flexibility to either begin use the revised surveys in fall 2025, or to wait until Jan. 1, 2026. Those grantees who begin administering the revised surveys in fall 2025 will submit those data during the Jan./Feb. 2026 data submission. Grantees who wait until Jan. to begin using the revised surveys will first submit data from them in summer 2026.

Any questions on these plans, before we dive into suggestions for revisions?

PREP Participant Entry and Exit Surveys

Measures of Participant Characteristics, Behaviors, Program Experiences, and Perceptions of Program Effects

•Entry survey

- –Characteristics
- –Behaviors

As a reminder, data from the participant entry and exit surveys are used to describe the youth served by PREP programs; youth perceptions of program effects on their sexual intentions, knowledge, attitudes, and beliefs related to topics covered by PREP; and their experiences in the program. We’ll talk through each of these sections today, and then at the end of the session will open up the discussion for any broader comments that don’t fit into a particular section.

Participant Characteristics
(Entry and Exit Survey Questions 1-7)

Purpose: To obtain data on the characteristics of the youth who receive PREP programming

•Age
•Grade
•Language(s) spoken at home or with family
•Hispanic/Latino ethnicity
•Race
•Sex
•Living situation

Both the entry and exit surveys ask identical questions about participant characteristics—specifically: age, grade, language(s) spoken at home, ethnicity, race, sex, and living situation. These measures are intended to obtain data on the characteristics of the youth who receive PREP programming.

The next few slides are going to focus on each of these in turn, but before we ask for suggestions on the existing questions, please think about the following:

•Are there any other demographic characteristics you wish were captured in the performance measures data but are not currently?
•[If any, probe: What do others think about that suggestion? Would having data on that be useful to you? Worth the additional burden of making the survey a bit longer?]

[These questions are asked in both middle and high school versions of the surveys at entry and exit.]

Age (Survey Question 1)

•Question/instructions:

How old are you?

MARK ONLY ONE ANSWER

•Response options:

Middle school version

- •10-16

Although this question is asked in both middle and high school versions of the surveys, the response categories differ by version. All responses are numeric, and the options are 10, 11, 12, etc. up to 16 in the middle school version and up to 20 in the high school and older version

•Are the youth you serve able to understand and answer this question?

- •(If not: What part is problematic? What changes would you suggest?)
•Should we change the lower end of the range in the high school+ version from 10 to an older age? [If so, what age would you expect to be the youngest students in high schools in your communities?]
•Do you have any other suggestions for improvements to this question?
- •[If no comments: This question is pretty straightforward, so let’s move on to others.]

Grade (Survey Question 2)

•Response options: Middle school version

- - - - •5th-9th
      - •My school does not assign grade levels
      - •I am not currently enrolled in school

Grade is another question for which the response categories differ in the middle and high school versions of the surveys. In each version, there are some numeric options, plus a few options when a numeric grade does not apply. The numeric options are 5th through 9th in the middle school version and 9th through 12th in the high school and older version.

Are the youth you serve able to understand and answer this question?

•[probe as needed:]

- •Do they understand what the question is asking? (If not: What part is unclear? What changes would you suggest?)
- •Do youth have any trouble understanding any of the response categories? (If so: What wording would you suggest?)
•Do the response options adequately cover all the situations in your sites? Are there any other response options that would be helpful for your populations?
•Do you have any other suggestions for improvements to this question?
•[if no comments, could ask: Age and grade align to some extent, but not perfectly. Do you think it’s useful to measure both in the performance measures data? (If anyone says no, ask: Which do you think is more important/more useful to you?]

Language (Survey Question 3)

•Question/instructions:

When you are at home or with your family, what language or languages do you usually speak?

MARK ALL THAT APPLY

•Response options:

- •English
- •Spanish
- •Other (specify)

•Are the youth you serve able to understand and answer this question?

- •(If not: What part is problematic? What changes would you suggest?)
- •Do you have any other suggestions for improvements to this question?

•[If no comments: This question is pretty straightforward, so let’s move on to others.]

Race, Ethnicity, and Sex (Survey Questions 4-6)

•Race/Ethnicity -- to be revised based on new OMB guidance

- •American Indian or Alaska Native
- •Asian
- •Black or African American
- •Hispanic or Latino
- •Middle Eastern or North African
- •Native Hawaiian or Pacific Islander
- •White
•Sex – to be revised based on results of cognitive pre-test

Unlike the other slides, this slide does not show the wording from the current PREP surveys, because the measures listed on this slide are definitely going to change.

As I mentioned earlier, we plan to revise the race/ethnicity questions to align with recent changes at OMB [and the Census Bureau], which are being implemented across government agencies. The guidance is for one mark-all-that-apply question with the categories shown on the screen.

[pause briefly, but don’t specifically ask for input]

[if asked: Key differences between the old race/ethnicity items and the new guidance are: combining race ethnicity into a single “mark all that apply” question (with “Hispanic or Latino” as a category) and including a new category for “Middle Eastern or North African”]

Similarly, ACF is conducting a set of cognitive pre-tests of new items on sexual orientation and gender identity (SOGI), testing questions recommended by the National Academies of Sciences, Engineering, and Medicine (The National Academies, NAS) and the Office of Management and Budget.
[If asked, NASEM in particular conducted a thorough literature review and consultation sponsored by NIH. While the measures in the youth space continue to evolve, the measures were highly recommended but have not been uniformly test with youth. Testing with youth across ACF programs will occur in early 2025.]

The results of those pretests will likely drive the revisions to those measures, so we are not specifically soliciting your comments on these items. [pause in case anyone comments anyway]

[if asked: The SOGI pretests will involve more complex items than the binary sex question currently in the surveys, such as a pair of questions on sex assigned at birth and gender identity, and a separate question on sexual orientation identity.]

Living Situation (Survey Question 7)

•Response options:

- •Living with family
- •In foster care, living with a family
- •In foster care, living in a group home
- •Couch surfing or moving from home to home
- •Living outside, in a tent city or homeless camp, in a car, in an abandoned vehicle or in an abandoned building

Are the youth you serve able to understand and answer this question?

•[probe as needed:]

- •Do they understand what the question is asking? (If not: What part is unclear? What changes would you suggest?)
- •Do youth have any trouble understanding any of the response categories? (If so: What wording would you suggest?)
•[If it comes up:] The purpose of this question is to obtain information on three populations of particular importance to FYSB: runaway/homeless youth(RHY), adjudicated youth, and foster care populations. When developing the current categories, ACF worked hard to break down those three populations down into sub-items that youth would understand.]
•Any other comments on this question?

Participant Behaviors
(Entry Survey Questions 8-18)

Question number	Topics	Survey version
8-12	Adulthood preparation subjects	MS and HS+
13-18	Sexual risk behaviors	HS+ only

For State/Competitive PREP sessions only

The entry surveys include questions about youth behaviors related to Adulthood Preparation Subjects

and sexual risk behaviors prior to beginning the PREP program. We’ll go through each of these in the next slides.

[Both the middle school and the high school and older versions of the surveys include questions related to Adulthood Preparation Subjects. The high school and older version of the entry survey also includes questions about sexual intercourse and potential consequences.]

Entry Survey Question 8

Question number	Item	Response options
8a	resisted or said no to peer pressure?	All of the time	Most of the time	Some of the time	None of the time
8b	managed your emotions in healthy ways (for example, ways that are not hurtful to you or others)?	All of the time	Most of the time	Some of the time	None of the time
8c	made decisions to not use drugs and alcohol?	All of the time	Most of the time	Some of the time	None of the time
8d	thought about the consequences before making a decision?	All of the time	Most of the time	Some of the time	None of the time

For State/Competitive PREP sessions only

Question 8 focuses on ….

•Are the youth you serve able to understand and answer these questions?
•[probe as needed:]

- •Do they understand what the questions are asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
- •How about the instructions—do youth have any trouble understanding that part? (If so: What wording would you suggest?)
•Do the youth you serve have any trouble understanding the response options?
- •For example, do they understand the difference between “most of the time” and “some of the time”?
- •Would the question be easier for youth to answer if the middle two categories were combined? if we dropped “most of the time” but kept “some of the time”?
- •Any other suggestions related to the response options?

•These questions are included in both the middle school and the high school and older versions of the surveys. Does that seem appropriate for these questions?
•Any other comments on these items?

Entry Survey Question 9

Question number	Item	Response options
9a	I make plans to reach my goals.	Not true at all	Somewhat true of me	Very true of me
9b	I care about doing well in school.	Not true at all	Somewhat true of me	Very true of me
9c	I plan to graduate high school or get my GED.	Not true at all	Somewhat true of me	Very true of me
9d	I plan to get more education and/or training after high school or completing my GED.	Not true at all	Somewhat true of me	Very true of me
9e	I plan to get a steady full-time job after school.	Not true at all	Somewhat true of me	Very true of me
9f	I would speak up or ask for help if I was being bullied in person or online, via text, while gaming, or through other social media.	Not true at all	Somewhat true of me	Very true of me
9g	I would speak up or ask for help if others were being bullied in person or online, via text, while gaming, or through other social media	Not true at all	Somewhat true of me	Very true of me

For State/Competitive PREP sessions only

•Are the youth you serve able to understand and answer these questions?
•[probe as needed:]

- •Do they understand what the questions are asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
- •Do youth have any trouble understanding the response options? (If so: What wording would you suggest?)
- •How about the instructions—do youth have any trouble understanding that part? (If so: What wording would you suggest?)

•These questions are included in both the middle school and the high school and older versions of the surveys. Does that seem appropriate for these questions?
•Any other comments on these items?

[for a later slide: … would it be better to have more even… even if that meant different topics together?]

Entry Survey Question 10

Question number

Item

Response options

10a

I save money to get things I want.

Not true

at all

Somewhat

true of me

Very true

of me

10b

I feel confident about how to open a bank account.

Not true

at all

Somewhat

true of me

Very true

of me

10c

I feel confident about how to prepare a budget.

Not true

at all

Somewhat

true of me

Very true

of me

10d

I feel confident about how to track my expenses.

Not true

at all

Somewhat

true of me

Very true

of me

10e

I understand the costs associated with raising a child.

Not true

at all

Somewhat

true of me

Very true

of me

For State/Competitive PREP sessions only

Question 10 has the same question stem, instructions, and response options as Q9, which we just discussed, but the items in the five rows are different—relating to financial literacy--so let’s focus on them:

•Are the youth you serve able to understand and answer these questions?
•[probe as needed:]

- •Do they understand what the questions are asking? (If not: What part is unclear? What changes would you suggest to the current wording?)

•These questions are included in both the middle school and the high school and older versions of the surveys. Does that seem appropriate for these questions?
•Any other comments on these items?

Entry Survey Question 11

Question number	Item	Response options
11a	talked with your parent, guardian, or caregiver about things going on in your life?	All of the time	Most of the time	Some of the time	None of the time
11b	talked with your parent, guardian, or caregiver about sex?	All of the time	Most of the time	Some of the time	None of the time

For State/Competitive PREP sessions only

Question 11 has the same question stem, instructions, and response options as Q8, which we already discussed, but the items in the four rows are different, so let’s focus on them:

•Are the youth you serve able to understand and answer these questions?
•[probe as needed:]

- •Do they understand what the questions are asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
•I noted that Questions 8 and 11 have the same question stem, instructions, and response options. Do you think it would be easier for youth to follow if we put those questions next to each other [rather than having questions with different response options in between them]? [How about combining 8 and 11 into one table, with six rows, instead of two separate tables with two to four rows each? Even though they cover different topics?]

•These questions are included in both the middle school and the high school and older versions of the surveys. Does that seem appropriate for these questions?
•Any other comments on these items?

Entry Survey Question 12

Question number	Item	Response options
12a	I understand what makes a relationship healthy.	Not true at all	Somewhat true of me	Very true of me
12b	I would be able to resist or say no to someone if they pressured me to participate in sexual acts, such as kissing, touching private parts, or sex.	Not true at all	Somewhat true of me	Very true of me
12c	I would talk to a trusted person/adult (for example, a family member, teacher, counselor, coach, etc.) if someone makes me uncomfortable, hurts me, or pressures me do things I don’t want to do.	Not true at all	Somewhat true of me	Very true of me

For State/Competitive PREP sessions only

Question 12 has the same question stem, instructions, and response options as Qs9 and 10, which we already discussed, but the items in the three rows are different, so let’s focus on them:

•Are the youth you serve able to understand and answer these questions?
•[probe as needed:]

- •Do they understand what the questions are asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
•Given that this questions had the same question stem, instructions, and response options as Questions 9 and 10, do you think it would be easier for youth to follow if we put those questions next to each other [rather than having a question with different response options in between them]?
- •[if yes—and if we have time—could ask: The three questions with similar structures have very different numbers of rows. Q9 has seven rows, while this one has just three. Do you think the table questions would be easier for youth if the numbers of rows didn’t vary so much? For example, would moving the two items on bullying from Q9 to Q12 make Q9 less overwhelming? Or would they not fit as well given the topics covered?]

•These questions are included in both the middle school and the high school and older versions of the surveys. Does that seem appropriate for these questions?
•Any other comments on these items?

Sexual Intercourse (Entry Survey Questions 13-14)

•Question 13:

Have you ever had sexual intercourse?

MARK ONLY ONE ANSWER

- •Yes
- •No
•Question 14:

During the past 3 months, with how many people did you have sexual intercourse?

MARK ONLY ONE ANSWER

- •I have never had sexual intercourse
- •I have had sexual intercourse, but not in the past 3 months
- •1 person
- •2-3 people
- •4 or more people

For State/Competitive PREP sessions only

Starting with Q13,

•Are the youth you serve able to understand and answer this question?

- •(If not: What part is problematic? What changes would you suggest?)
- •Do you have any other suggestions for improvements to this question?
•[If no comments: This question is pretty straightforward, so let’s move on to Q14.]

Q14:

•Are the youth you serve able to understand and answer Question 14?
•[probe as needed:]
- •Do they understand what the question is asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
- •Do youth have any trouble understanding the response options? (If so: What wording would you suggest?)

- - •[The first two response options are very different from the last three. Is that confusing to youth? Would it be easier if we changed the order of the response options, so the ones with #s of people came first [before the ones that implicitly mean 0]?
  - •[Would it be clearer if we combined the first two response options?

- - - •Into “I have not had sexual intercourse in the past 3 months”
    - •Or would wording like “0 people” work better, since it’s more parallel with the later three response options?]

[internal note: Before grantees began submitting individual-level survey data, we needed two separate categories; but now that we have individual-level survey data, we can use Q13 to distinguish between the two.

- •[internal note: We could instruct youth who responded “no” to Q13 to skip Q14-Q17, but skip logic can be confusing so we tried to avoid it here.]

- •How about the instructions—do youth have any trouble understanding that? (If so: What wording would you suggest?)

•These questions are included in only the high school and older version of the survey (not the middle school version). Does that seem appropriate for these questions?
•Any other comments on these items?

Condoms and Birth Control (Entry Survey Questions 15-16)

•Question 15:

If you had sexual intercourse in the past 3 months, how often did you or a partner use a condom?

MARK ONLY ONE ANSWER

For State/Competitive PREP sessions only

Questions 15 and 16 are related and have the same response options. Let’s talk about the question parts 1st, and then the responses. Starting with Q15,

•Are the youth you serve able to understand and answer this question?
•[probe as needed:]

- •Do they understand what the question is asking? (If not: What part is unclear? What changes would you suggest to the current wording?)

Q16,

•How about the wording of Q16, are the youth you serve able to understand this question?

•[probe as needed:]
- •Do they understand what the question is asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
- •How about the definition in italics? Is that clear? (If not: What part is unclear? What changes would you suggest to the current wording?)

both: Turning now to the response options,

•Do youth have any trouble understanding the response options? (If so: What wording would you suggest?)
- •[As we discussed on the previous slide, we could combine the first two response options, if that would be easier for youth.]
- •[As we discussed on an earlier question, do youth understand the difference between “most of the time” and “some of the time”? Would the question be easier for youth to answer if those two categories were combined—e.g. if we dropped “most of the time” but kept “some of the time”?]

•These questions are included in only the high school and older version of the survey (not the middle school version). Does that seem appropriate for these questions?
•Any other comments on these items?

Pregnancy and STIs (Entry Survey Questions 17-18)

•Question 17:

To the best of your knowledge, have you ever been pregnant or gotten someone else pregnant?

MARK ONLY ONE ANSWER

- •I have never had sexual intercourse
- •Yes
- •No
- •Not sure
•Question 18:

Have you ever been told by a doctor or other medical provider that you had a sexually transmitted infection (STI)?

MARK ONLY ONE ANSWER

- •Yes
- •No

For State/Competitive PREP sessions only

Starting with Q17,

•Are the youth you serve able to understand and answer this question?
•[probe as needed:]

- •Do they understand what the question is asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
- •Do youth have any trouble understanding the response options? (If so: What wording would you suggest?)
- •Do you have any other suggestions for improvements to this question?

Q18:

•Are the youth you serve able to understand and answer Question 18?
•[probe as needed:]
- •Do they understand what the question is asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
- •Do youth have any trouble understanding the response options? (If so: What wording would you suggest?)
- •Do you have any other suggestions for improvements to this question?

•These questions are included in only the high school and older version of the survey (not the middle school version). Does that seem appropriate for these questions?
•Any other comments on these items?

Participant Perceptions
(Exit Survey Questions 8-17)

Question number	Topics	Survey version
8-12	Perceptions of program effects on behaviors related to adulthood preparation subjects	MS and HS+
13-15	Perceptions of program effects on sexual risk behaviors	HS+ only
16-17	Perceptions of PREP programming	MS and HS+

Now let’s turn to the exit survey. In addition to participant characteristics, which we discussed already, the participant exit surveys include measures of youth perceptions of program effects on their decision to engage in behaviors related to the APS. The high school and older version of the exit survey also includes measures of youth perceptions of program effects on their decision to engage in sexual risk behaviors.

[The Q#s shown in the table refer to the HS+ version of the surveys. Q16-17 in that version are numbered Q13-14 in the MS version.]

Exit Survey Questions 8-12

•Question/instructions:

Even if your program didn’t cover this topic would you say that being in the program made you more likely, about the same, or less likely to… (Note: If the program has not affected your likelihood to do the following, choose “About the same”.)

MARK ONLY ONE ANSWER PER ROW

•Sub-question wording similar to entry survey items
•Response options:

- oMuch more likely
- oSomewhat more likely
- oAbout the same
- oSomewhat less likely
- oMuch less likely

For State/Competitive PREP sessions only

Exit survey Questions 8-12 all use the same question stem and response categories, shown on this slide.

In addition, the wording of most of the subquestions are identical to questions in the entry survey that we just discussed. We will apply any wording changes made in the entry survey to the similar items in the exit survey.

So we’re not going to talk through all of the exit survey questions in detail today. However, we will talk briefly through Question 8, as an example, on the next slide.

But before we turn to that example, I’d like to ask for any general comments on the question stem, instructions, or response categories used for questions 8-12:

•Are the youth you serve able to understand the question stem and instructions used for these questions? (If not: What part is unclear? What changes would you suggest to the current wording?)
•How about the response categories? Do youth have any trouble understanding them?

- •For example, do they understand the difference between “much” and “somewhat”?
- •Would the question be easier for youth to answer if we revised the response options? (If so: What changes would you suggest? For example, would it be clearer if there were only three response options: "More likely", "About the same“, and "Less likely"?))
- •Any other suggestions related to the response options?

•[If anyone asks why the exit survey can’t ask the same questions as the entry survey, so they can be compared as in a pre/post design: Assessing pre/post differences without a comparison group would be inappropriate due to maturation effect. The entry and exit surveys are deliberately designed to not include identical questions on behaviors, to discourage potentially misleading comparisons. Instead, the performance measures assess behaviors at entry, and perceptions on how participants think they will behave in the future at exit.]

Exit Survey Question 8

Question number	Item	Response options
8a	resisted or said no to peer pressure?	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely
8b	managed your emotions in healthy ways (for example, ways that are not hurtful to you or others)?	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely
8c	made decisions to not use drugs and alcohol?	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely
8d	thought about the consequences before making a decision?	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely

For Tribal PREP/PREIS sessions only: Starting with exit survey Question 8…

•Are the youth you serve able to understand and answer these questions?
•[probe as needed:]

- •Do they understand what the questions are asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
- •How about the instructions—do youth have any trouble understanding that part? (If so: What wording would you suggest?)
•How about the response options? Do youth have any trouble understanding them?
- •For example, do they understand the difference between “much” and “somewhat”?
- •Would the question be easier for youth to answer if we revised the response options? (If so: What changes would you suggest? For example, would it be clearer if there were only three response options: "More likely", "About the same“, and "Less likely"?)
- •Any other suggestions related to the response options?

•These questions are included in both the middle school and the high school and older versions of the surveys. Does that seem appropriate for these questions?
•Any other comments on these items?

Exit Survey Question 9

Question number	Item	Response options
9a	make plans to reach my goals.	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely
9b	care about doing well in school.	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely
9c	graduate high school or get my GED.	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely
9d	get more education and/or training after high school or completing my GED.	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely
9e	get a steady full-time job after school.	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely

For Tribal PREP/PREIS sessions only: Now let’s look at exit survey Question 9. Question 9 has the same question stem, instructions, and response options as Q8, which we just discussed, but the items in the # rows are different, so let’s focus on them:

•Are the youth you serve able to understand and answer these questions?
•[probe as needed:]

- •Do they understand what the questions are asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
•These questions are included in both the middle school and the high school and older versions of the surveys. Does that seem appropriate for these questions?
•Any other comments on these items?

Exit Survey Question 10

Question number	Item	Response options
10a	save money to get things I want.	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely
10b	feel confident about how to open a bank account.	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely
10c	feel confident about how to prepare a budget.	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely
10d	feel confident about how to track my expenses.	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely
10e	understand the costs associated with raising a child.	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely

For Tribal PREP/PREIS sessions only: Now let’s look at exit survey Question 10. This question has the same stem, instructions, and response options as Q8-9, so let’s focus on the items in the # rows:

•Are the youth you serve able to understand and answer these questions?
•[probe as needed:]

- •Do they understand what the questions are asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
•These questions are included in both the middle school and the high school and older versions of the surveys. Does that seem appropriate for these questions?
•Any other comments on these items?

Exit Survey Question 11

Question number	Item	Response options
11a	talk with your parent, guardian, or caregiver about things going on in your life?	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely
11b	talk with your parent, guardian, or caregiver about sex?	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely

For Tribal PREP/PREIS sessions only: Now let’s look at exit survey Question 11. This question has the same stem, instructions, and response options as Q8-10, so let’s focus on the items in the # rows:

•Are the youth you serve able to understand and answer these questions?
•[probe as needed:]

- •Do they understand what the questions are asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
•These questions are included in both the middle school and the high school and older versions of the surveys. Does that seem appropriate for these questions?
•Any other comments on these items?

Exit Survey Question 12

Question number	Item	Response options
12a	I understand what makes a relationship healthy.	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely
12b	I would be able to resist or say no to someone if they pressured me to participate in sexual acts, such as kissing, touching private parts, or sex.	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely
12c	I would talk to a trusted person/adult (for example, a family member, teacher, counselor, coach, etc.) if someone makes me uncomfortable, hurts me, or pressures me do things I don’t want to do.	Much more likely	Somewhat more likely	About the same	Somewhat less likely	Much less likely

For Tribal PREP/PREIS sessions only: Now let’s look at exit survey Question 12. This question has the same stem, instructions, and response options as Q8-11, so let’s focus on the items in the # rows:

•Are the youth you serve able to understand and answer these questions?
•[probe as needed:]

- •Do they understand what the questions are asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
•These questions are included in both the middle school and the high school and older versions of the surveys. Does that seem appropriate for these questions?
•Any other comments on these items?

Exit Survey Question 13

•Question/instructions:

As a result of being in the program, are you planning to abstain from sexual intercourse (choose to not have sexual intercourse) for at least the next 3 months?

•Response options:

- oYes GO TO QUESTION 14
- oNo GO TO QUESTION 15, NEXT PAGE
- oNot sure GO TO QUESTION 15, NEXT PAGE

Question 13 is one of a set of questions included in the high school and older version of the exit survey that aren’t in the middle school version, or in the entry survey.

Q13 measures whether, as a result of the program, respondents are planning to abstain from sexual intercourse. Depending on their response to this question, youth are instructed to either go to the next question (Q14) or to skip over it and go to Q15.

•Are the youth you serve able to understand and answer this question?
•[probe as needed:]

- •Do they understand what the question is asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
- •Do youth have any trouble understanding the response options? (If so: What wording would you suggest?)

•Do the youth you serve have any trouble understanding the instructions on whether to answer Q14 or skip to Q15? (If so: What wording would you suggest to make the skip instructions easier to follow?)
- •Any other ideas on the skip instructions?

•This question is included in only the high school and older version of the survey (not the middle school version). Does that seem appropriate for this question?
•Any other comments on this item?

Exit Survey Question 14

Question number	Item	Response options
14a	how it might affect your plans for the future	Not at all important	Not too important	Somewhat important	Very important
14b	the possible emotional and social consequences (for example, feeling sadness or regret, disappointing your parent(s) or guardian(s), and/or negative reactions from your peers)	Not at all important	Not too important	Somewhat important	Very important
14c	the risk of getting a sexually transmitted infection (STI)	Not at all important	Not too important	Somewhat important	Very important
14d	the risk of getting pregnant or getting someone pregnant.	Not at all important	Not too important	Somewhat important	Very important

Those who respond “yes” to Q13 are asked how important certain reasons are in their decision to abstain.

•Are the youth you serve able to understand and answer these questions?
•[probe as needed:]

- •Do they understand what the questions are asking?
- •How about the instructions—do youth have any trouble understanding that part? (If so: What wording would you suggest?)
•Do youth have any trouble understanding the response options? (If so: What changes would you suggest?)
- •Do you think it would be easier for youth to answer if there were only three response categories, instead of four? (If so: Which category(ies) would you dropping or combining?)

•These questions are included in only the high school and older version of the survey (not the middle school version). Does that seem appropriate for these questions?
•Any other comments on these items?

Transition to Exit Survey Question 15

•Instructions:

- –At the end of the page, after Questions 13 and 14

IF YOU ANSWERED “YES” TO QUESTION 13, GO TO QUESTION 16 IF YOU ANSWERED “NO” OR “NOT SURE” TO QUESTION 13: SKIP QUESTION 14 AND GO TO QUESTION 15 ON THE NEXT PAGE

- –At the beginning of the next page, just before Question 15

The next few questions refer to sexual intercourse and your risk of pregnancy and sexually transmitted infections (STIs). Remember, all of your responses will be kept private. (Note: Do not answer this question if you responded “Yes” to question 13.)

Exit Survey Question 15a

•Question/instructions:

Has being in the program made you more likely, about the same, or less likely to… (Note: If the program has not affected your likelihood to do the following, choose “About the same”.)

a. have sexual intercourse in the next 3 months?

MARK ONLY ONE ANSWER

•Response options:

- oMuch more likely
- oSomewhat more likely
- oAbout the same
- oSomewhat less likely
- oMuch less likely

Question15 has three subquestions, and this slide shows the first one, along with the question stem, instructions, and response categories.

The question stem and response categories are the same as those used for exit survey Questions 8-12 , which we already discussed.

So rather than talking through those again, let’s focus on the wording of the subquestions. This slide shows Q15 subquestion a.

•Are the youth you serve able to understand and answer this question?
•[probe as needed:]

- •Do they understand what the question is asking? (If not: What part is unclear? What changes would you suggest to the current wording?)
•This question is included in only the high school and older version of the survey (not the middle school version). Does that seem appropriate for this question?
•Any other comments on this item?

Exit Survey Question 15b-c

•Subquestions:

b. use (or ask your partner to use) a condom if you were to have sexual intercourse in the next 3 months?

c. use (or ask your partner to use) birth control OTHER than condoms if you were to have sexual intercourse in the next 3 months? By birth control, we mean methods that can prevent pregnancy, like using birth control pills, the shot, the patch, the ring, IUD, or implant.

• Instructions for each:

MARK ONLY ONE ANSWER

•Same response options as Question 15a

This slide shows the other two subquestions of Question 15. The wording of these subquestions is similar to that used in Entry Survey Questions 15-16, and we already talked through the question stem, instructions, and response categories. We’ll apply any changes to those consistently across these subquestions as well, so unless anyone has anything to add that hasn’t been raised already, we’ll move on to the next question.

Exit Survey Question 16

Question number	Item	Response options
16a	did you feel interested in program sessions and classes?	All of the time	Most of the time	Some of the time	None of the time
16b	did you feel the material presented was clear?	All of the time	Most of the time	Some of the time	None of the time
16c	did discussions or activities help you to learn program lessons?	All of the time	Most of the time	Some of the time	None of the time
16d	did you have a chance to ask questions about topics or issues that came up in the program?	All of the time	Most of the time	Some of the time	None of the time
16e	did you feel respected as a person?	All of the time	Most of the time	Some of the time	None of the time

At the end of the exit survey includes questions related to youth program experiences. Q16 includes five items with the same question stem: [“Even if you didn’t attend all of the sessions or classes in this program, how often in this program…”]

•Are the youth you serve able to understand and answer these questions?
•[probe as needed:]

- •Do youth have any trouble understanding this question stem or instructions? (If so: What wording would you suggest?)
- •How about the five sub-items? Do youth have any trouble understanding any of these? (If so: What wording would you suggest?)
- •[The four response options are the same as the response options are used in several of the entry survey questions, so we don’t need to talk through those again. We’ll apply any changes to those options consistently across the questions that use them.]

[the Question number depends on the survey version: Q13 in MS version, Q16 in HS and older version

Exit Survey Question 17

Question number	Item	Response options
17a	the amount of information you received about abstaining from sex (choosing to not have sex)?	Very satisfied	Somewhat satisfied	A little satisfied	Not at all satisfied
17b	the amount of information you received about condoms and birth control?	Very satisfied	Somewhat satisfied	A little satisfied	Not at all satisfied

Finally, the exit survey Q17 includes a pair of items about satisfaction with program content.

•Are the youth you serve able to understand and answer these questions?
•[probe as needed:]

- •Do youth have any trouble understanding this question stem or instructions? (If so: What wording would you suggest?)
- •How about the two sub-items? Do youth have any trouble understanding any of these? (If so: What wording would you suggest?)
- •Do youth have any trouble understanding the response options? (If so: What wording would you suggest?)

•Are there any other aspects of program experience that you wish were captured in the performance measures data but are not currently?
•Any other comments on this section?

[If asked: the Question number depends on the survey version: Q14 in MS version, Q17 in HS and older version]

ADDITIONAL SUGGESTIONS AND COMMENTS

Now that we’ve talked through each of the existing sections in the participant entry and exit surveys, I’d like to broaden the conversation to any comments that don’t fit into a particular section.

•Do you have any other suggestions for improvements to the surveys?

•Are there any other topics that you wish were captured in the performance measures data but are not currently?
•Any suggestions for improvements to the instructions in the surveys?
•Any other comments on the participant entry and exit surveys?
•[if we have time left at the end: How about other performance measures, beyond the surveys? Any comments on the measures of attendance, reach, and dosage? Any comments on the measures of structure, cost, and support?]

•[If anyone suggests linking the entry and exit surveys to use as a pre/post design: Assessing pre/post differences without a comparison group would be inappropriate due to maturation effect. The entry and exit surveys are deliberately designed to not include identical questions on behaviors, to discourage potentially misleading comparisons. Instead, the performance measures assess behaviors at entry, and perceptions on how participants think they will behave in the future at exit.]

Next Steps

•Input gathering – fall 2024

- oIf you have any additional comments, please share them via the help desk at P[email protected]
•Preliminary revisions – winter 2025
•OMB review – spring/summer 2025
•Introduce approved revisions – September 2025
•Begin administering revised surveys:

- oOptional – fall 2025
- oRequired for all PREP grant recipients – Jan. 1, 2026
•Submit data from revised surveys:
- oOptional – Jan./Feb. 2026
- oRequired for all PREP grant recipients – July/Aug. 2026

We’re here to help!

For more support, contact the Mathematica

PREP Performance Measures technical assistance team.

[email protected]

1-855-267-6270 (toll-free)

Thank you all for participating on today’s listening session. The last slide shows how you can reach the help desk, if you have any additional thoughts later, or any questions about the performance measures at any time.

File Type	application/vnd.openxmlformats-officedocument.presentationml.presentation
File Modified	0000-00-00
File Created	0000-00-00