Supporting Statement A 0990-0392 Renewal_OAH

Supporting Justification for the Extension of OMB Clearance of the

“Office of Adolescent Health and Administration for Children, Youth and Families Teen Pregnancy Prevention Performance Measure Collection”

Part A: Justification for the Collection of the Data

April 2015

Submitted by:

Office of Adolescent Health

U.S. Department of Health and Human Services

1101 Wootton Parkway, Suite 700

Rockville, MD 20852

Project Officer: Amy Farb

INTRODUCTION

The Office of Adolescent Health (OAH) is requesting an extension through December 2015 of the OMB # 0990-0392 “Office of Adolescent Health and Administration for Children, Youth and Families Teen Pregnancy Prevention Performance Measure Collection,” currently approved through 5/31/2015. There are no additions and only minor changes to the data that will be collected; the only changes to the data collection are some reductions in the data collected. The reductions include both fewer grantees reporting on some of the measures, and data being collected on fewer measures —as specified further on in this document. 

A1. Circumstances Making the Collection of Information Necessary

1. Background

In 2010, OAH and FYSB awarded 5-year grants for teen pregnancy prevention under TPP and PREIS, respectively. Currently, TPP is funding 94 grantees, and PREIS is funding 11 grantees (Exhibit 1). Of the 94 TPP grantees, 77 are “Tier 1”—grants to replicate programs that have already been proven effective to reduce teenage pregnancy as identified through the HHS Teen Pregnancy Prevention Evidence Review managed by the Assistant Secretary for Planning and Evaluation (ASPE). Interventions for these programs vary widely in terms of duration (from 1 day to 4 years), setting (schools, clinics, or community based settings), populations served (middle school students, high school students, parents of teens) and content (e.g., youth development programs or sex education programs).

The remaining 18 TPP grants (“Tier 2”) and the 11 PREIS grants are research and demonstration grants to develop, refine, and test additional models and innovative strategies. Tier 2 and PREIS grants focus on areas with high teen pregnancy rates and high-risk, vulnerable, and culturally underrepresented youth populations, including youth in foster care, runaway and homeless youth, pregnant and parenting teens, youth living in areas with high teen birth rates, delinquent youth, and youth who are disconnected from usual service delivery systems.

The Tier 1 C/D and the Tier 2 and PREIS grantees are all required to conduct independent rigorous evaluations; Tier 1 A/B grantees are not.

Exhibit 1: Summary of current TPP and PREIS grants (2010-2015)

	Agency	Description	Independent rigorous evaluation	# of grants
TPP grants
Tier 1 A/B	OAH	Replication grants funded at <$1 million/year,	No	61
Tier 1 C/D	OAH	Replication grants funded at >$1 million/year	Yes	16
Tier 2	OAH	Research and demonstration grants	Yes	18
PREIS grants	FYSB	Research and demonstration grants	Yes	11
TOTAL				106

FY 2014 is the last year of funding for this cohort of TPP and PREIS grantees. They will need to report on their performance measures one more time, in the fall of 2015. This is after the current OMB clearance expires on 5/31/2015, so renewal clearance is being requested through December 2015.

b. Legal or Administrative Requirements that Necessitate the Collection

The TPP program, administered by OAH, was originally authorized under the Consolidated Appropriations Act, 2010 (P.L. 111-117) and currently operates under authority contained in Consolidated Appropriations Act, 2015. The Act provides $100,000,000 annually for making competitive contracts and grants to public and private entities to fund medically accurate and age appropriate programs that reduce teen pregnancy, and for the Federal cost associated with administering and evaluating such grants and contracts. The PREIS program, administered by FYSB, uses funds authorized by the Patient Protection and Affordable Care Act, 2010 (P.L. 111-148). The Act authorized FYSB to award $10 million in grants to entities to implement innovative youth pregnancy prevention strategies.

The performance measure collection provides OAH and FYSB with data to both effectively manage the grant programs, and to comply with accountability and federal performance requirements for the 1993 Government Performance and Results Act (P.L. 103-62). Collecting and reporting on data for performance measures are a funding requirement for the grants, as stated in the funding opportunity announcement.

c. Study Objectives

The purpose of this data collection is to collect performance measure data on the Teen Pregnancy Prevention (TPP) Program (administered by OAH) and the Personal Responsibility Education Program Innovative Strategies (PREIS) (administered by the Family and Youth Services Bureau (FYSB) of the Administration for Children and Families (ACF)). These data will allow OAH and FYSB to monitor the progress of program grantees, and to report to Congress on the performance of the programs. The respondents for this collection are grantees funded by the programs and in FYSB’s case, program participants.

The performance measures approved by OMB in 0990-0392 remain largely unchanged. The primary changes are reductions in the number of measures or the number of grantees reporting on the measures. Specifically:

• There remain 106 grantees but there are 2 fewer PREIS grantees and 2 additional OAH grantees.

• The perceived impact measures are reported only by PREIS grantees (as approved in our non-substantive change request from October 17, 2012) and there are now 11 PREIS grantees, down from the original 13 grantees.

• The number of youth participants in the PREIS grant programs were originally overestimated. There are approximately 2000 compared to the original 5,664.

• Some of the measures are no longer being collected. Specifically:

  • Measures of the soundness of evaluations

  • Measures of dosage for parents/other clients

  • Adherence to the program-specified number of sessions.

The performance measures to be reported by grantees are summarized in Exhibit 2.

Not all grantees will report on all measures:

• Measures to assess changes in participant behaviors (e.g., sexual activity, contraceptive use, condom use) or intentions (intention to have sex, use contraception, or use condoms) require a comparison group for meaningful interpretation. Only those grantees with rigorous evaluations (i.e., TPP Tier 1 C/D and Tier 2 and PREIS grantees) have data on both program participants and a comparison group; therefore, they are the only grantees to report data on these performance measures.

• PREIS grantees are the only ones to collect the perception of program impact measures.

• Research and demonstration grants (Tier 2 and PREIS) have a measure related to the packaging of their interventions so that they can be replicated in the future if found to be effective. Because Tier 1 grantees are implementing programs that have already been packaged for replication, this is not a relevant measure for them.

Exhibit 2: Measures reported by current grantees*

	Data source	Measures reported to OAH/FYSB
		Tier 1 A/B	Tier 1 C/D	Tier 2	PREIS
Participant-level measures
Behaviors and intentions (rigorous outcome data)
Any sex	Grantees’ evaluations		x	x	x
Condom use			x	x	x
Contraceptive use			x	x	x
Pregnancy			x	x	x
Intentions to have sex			x	x	x
Intentions to use condoms			x	x	x
Intentions to use contraception			x	x	x
Perception of program impact (post-test only data)
Perceived impact of program on sex	Grantees’ evaluations				x
Perceived impact of program on condom use					x
Perceived impact of program on contraceptive use					x
Perceived impact of program on abstinence					x
Dissemination
	Administrative records of grantees	x	x	x	x
# of manuscripts accepted for publication in journals			x	x	x
# of manuscripts submitted for publication			x	x	x
# of national, regional, or state-level presentations		x	x	x	x
% of Tier 2/PREIS programs that have completed development of pieces of program necessary to package it for replication (logic model, fidelity monitoring tools, etc.)				x	x
Reach and retention
# of youth served, by characteristics (e.g., age, gender, race/ethnicity, special populations)	Administrative records of grantees	x	x	x	x
# of parents/other clients served		x	x	x	x
# of formal and informal partners		x	x	x	x
Retention of partners		x	x	x	x
# of new facilitators trained		x	x	x	x
# of facilitators receiving follow-up training		x	x	x	x
Dosage
Mean and median percent of total intended program services received by youth	Grantee attendance records	x	x	x	x
% of youth who received at least 75% of the program		x	x	x	x
Fidelity
Adherence to program-specified activities, based on facilitator self-assessment	Fidelity monitoring logs	x	x	x	x
Adherence to program-specified activities, based on observations	Observation forms1	x	x	x	x
Quality of implementation		x	x	x	x
System in place to ensure fidelity	Fidelity process form2	x	x	x	x

* The four types of grantees represent different funding levels, resources and grant requirements. Therefore, as the table demonstrates, data is reported by grantee type.

A.2. Information Users

The data collection activities have provided OAH and FYSB leadership and program offices with information to more effectively manage the TPP and PREIS programs, respectively. The data have also been made available to members of Congress, the Office of Management and Budget, and the public at large to assess program performance. Grantees use this data to monitor and improve their programs. Continued use of these data is vital for ensuring on-going improvement of the TPP and PREIS programs and through dissemination efforts, broader understanding and support of programs designed to prevent teen pregnancy.

A.3. Use of Information Technology to Reduce Burden

Grantees enter performance measure data into a multi-use, Web-based reporting system. The Web-based system reduces burden for respondents by programming skip patterns, so that grantees only have to look at screens that are relevant for them. The programming automatically performs necessary calculations for respondents, and validates responses. A branching mode of presentation allows respondents to go directly to the sections they need, without having to go through the system in a linear progression. The system also automatically produces a data set, which saves time on preparation of the data for analysis.

A.4. Efforts to Identify Duplication

The OAH/FYSB performance measures data collection is the only data collection that provides information on the performance of the TPP and PREIS programs. The data collection makes use of existing data to the extent possible. For example, Tier 1 C/D, Tier 2, and PREIS grantees are already conducting rigorous evaluations of their programs. These grantees use findings from their evaluations to report on behavioral participant-level measures. Most of the additional measures are already being collected by grantees as part of their routine administrative records (e.g., numbers of publications, numbers of participants).

A.5. Impact on Small Businesses

No small businesses will be involved in the collection of data in this study.

A.6. Consequences of Not Collecting the Information/Collecting Less Frequently

GPRA requires that government agencies report on their performance measures annually. Therefore, it is essential that grantees report on these performance measures annually to OAH and FYSB. Collecting the data semi-annually is a critical management tool for OAH and FYSB staff to track the programs’ progress and take any corrective action that may be needed. In addition, collection and reporting of performance measure data is a requirement of all TPP and PREIS grantees as stated in the Funding Opportunity Announcement.

A.7. Special Circumstances

There are no special circumstances that occur when collecting this information.

A.8. Federal Register Comments and Persons Consulted Outside the Office of Adolescent Health

A 60-day notice was published in the Federal Register on XX/XX/2015, in Volume X, Number X, page XXX, and provided a 60-day period for public comments (Appendix A). No public comments were received.

OAH and FYSB consulted with staff of RTI International, the contractor responsible for assisting OAH and FYSB in developing the performance measures and performance measure reporting system, and a panel consisting of experts in the fields of performance measurement, teen pregnancy prevention, and evidence-based practice. In addition, OAH presented information on the performance measures to TPP and PREIS grantees and their evaluators at two conferences, and solicited their input. OAH also consulted and received feedback from other Federal staff working in the area of teen pregnancy prevention from ASPE, ACF, and CDC.

A list of individuals in the expert panel who provided input regarding the process evaluation is found in Exhibit 3.

Exhibit 3. Persons Consulted Outside the Agency

Expert Work Group

Donald Moynihan

[email protected]

University of Wisconsin

Lafollette School of Public Affairs

305 Observatory Hill Office Building

1225 Observatory Dr

Madison, WI 53706

(608) 263-6633

Kathryn Newcomer

[email protected]

George Washington

SPPPA

MPA Bldg 601

805 21st St NW

Washington, DC 20052

(202) 994-3959

Katherine Suellentrop
[email protected]
The National Campaign to Prevent Teen and Unplanned Pregnancy
1776 Massachusetts Ave, NW, suite 200
Washington, DC 20036
(202) 478-8515

Edward Mullen

[email protected]

Columbia University

School of Social Work

1255 Amsterdam Ave Room 1102

New York, NY 10027

(212) 851 2413

Douglas Kirby

[email protected]

ETR Associates

4 Carbonero Way

Scotts Valley, CA 95066

(831) 438-4060

Forrest Alton

[email protected]

SC Campaign to Prevent Teen Pregnancy

1331 Elmwood Avenue, Suite 140

Columbia, SC 29201

(803) 771-7700

Emily Ball

[email protected]

Administration for Children and Families

26 Federal Plaza

New York, NY 10278

(212) 264-2890 x273

A.9. Payments to Respondents

There are no payments to staff of grantee organizations completing the performance measure reporting form. For data collected from participants, grantees rolled the questions into questionnaires they are already using for enrollment and/or evaluation purposes to collect data from participants.

A.10. Assurance of Confidentiality

All data are reported de-identified. Grantees create their own unique randomly assigned identification numbers prior to entering any data.

The Web-based reporting system has also been designed to ensure the security of the data obtained. Individual grantee users have been assigned user names and passwords that grant them access to the project website. On the website, users provide data that are stored in a secure Microsoft SQL Server database utilizing a relational table structure, facilitating expedient data retrieval and analysis. The database server, located at RTI, is accessible only to the analysts assigned to this project. Electronic communications occur via a secure Internet connection. All transmissions are encrypted with 128-bit encryption through secure socket layers (SSL) and verified by a VeriSign^®, the leading SSL Certificate authority.

Data are stored on a secure, password-protected computer shared drive. All data files on multi-user systems will be under the control of a database manager, with access limited to project staff on a “need-to-know” basis only. No respondent identifiers are contained in reports generated by RTI, and results/data are only presented in aggregate form.

A.11. Justification for Sensitive Questions

The primary objective of the TPP and PREIS programs is to prevent teen pregnancy. The programs do this by promoting a decrease in sexual activity and/or an increase in contraceptive use. Because this is the primary focus of the programs, questions for the programs’ performance measures are necessarily related to these sensitive issues. Grantees with a rigorous evaluation already ask program participants (and adolescents in a comparison group) about sexual activity, contraceptive and condom use, as well as about intentions to have sex, use contraception, and use condoms. All of these data are de-identified.

The only sensitive questions that adolescents are asked specifically for this data collection –and only by PREIS grantees (not TPP grantees)—are four questions about their perception of the program’s impact on their behaviors:

1. Would you say that being in [NAME OF PROGRAM] has made you more or less likely to have sexual intercourse in the next year?

2. Would you say that being in [NAME OF PROGRAM] has made you more or less likely to abstain from sexual intercourse in the next year?

3. If you were to have sexual intercourse in the next year, would you say that being in [NAME OF PROGRAM] has made you more or less likely to use any of these methods of birth control?

• Condoms

• Birth control pills

• The shot (Depo Provera)

• The patch

• The ring (NuvaRing)

• IUD (Mirena or Paragard)

• Implant (Implanon)

4. If you were to have sexual intercourse in the next year, would you say that being in [NAME OF PROGRAM] has made you more or less likely to use a condom?

These data are also de-identified.

Neither OAH nor FYSB permit grantees to collect sexual activity data from very young client populations (sixth grade or under) nor, in the case of a school-based project, when a school board or district is opposed to the collection of these data.

In addition, PREIS grantees inform individual respondents that their participation is voluntary and that they may refuse to answer any or all of the questions in the instrument. Participants are also informed of the measures taken to protect the privacy of their responses.

A.12 Estimates of Annualized Burden Hours and Costs

a. Annualized Burden Estimates

The total annual burden is estimated to be 987 hours - 820 hours for grantees (to collect, summarize, and report the data for the performance measures) and 167 hours for PREIS program participants to respond to the survey questions about perceived impact of the program. The burden for the perceived impact questions includes PREIS youth in 7^th grade or higher.

Average burden hours for grantees

The TPP grantees will report performance measures for the last time in September 2015; PREIS grantees will report their performance measures for the last time in November 2015. Grantees will be collecting all of the data required for the performance measures as part of their administrative record-keeping or rigorous evaluation studies, so the only additional burden to grantees for reporting the performance measures is the time it takes them to assemble the necessary data and enter it into the reporting forms. Grantees can either upload spreadsheets into the web-based system or enter the data directly into the system using a point and click method.

• Reach and retention. Grantees report on measures of reach and retention. The reach data indicate the number of participants, by different background factors, the program is reaching. These data are based on basic demographic information that grantees collect on program participants when they are enrolled in the program. Grantees also report on the number of partners they are working with, partners retained, and the number of facilitators trained. Grantees collect these data for their own administrative purposes, and many have their own systems in place to track the data. We estimate that it takes each grantee approximately 4 hours to summarize and report these data.

• Dissemination and dosage. Grantees report on measures of dissemination and dosage. We estimate that this takes each grantee approximately 1 hour to summarize and report these data.

• Fidelity. Grantees collect several types of data related to fidelity as part of their ongoing administration of their programs. These include measures of adherence and quality, based on observations of a sample of sessions; a measure of adherence based on self-assessment forms completed by session facilitators; and a process measure of fidelity assessing the extent to which grantees have the necessary processes in place to ensure fidelity, to be completed by the grantee staff. We estimate that it takes grantees approximately 2 hours to summarize and report these data.

Tier 1 C/D and Tier 2/PREIS grantees report on seven additional participant-level measures that the Tier 1 A/B grantees do not report:

• Participant-level measures. Collection of the data and data entry are not an additional burden, because the grantees are collecting and entering these data as part of their evaluations. However, the time to create a dataset that contains only the variables needed for the performance measures in the required format is an additional burden. Grantees (or their evaluators) produce the required information with simple programming statements, so we estimate that it will take no more than 1 hour for each grantee to upload these data.

PREIS grantees report on perceived impact measures that TPP grantees do not report on.

• Perceived impact. For the perceived impact questions, FYSB PREIS grantees administer the questionnaires to program participants and enter the data into a database, which they then upload into the web portal. Questionnaires have a total of 10 questions—4 related to perceptions of impact, and 6 related to demographic characteristics of the respondents (age, grade, sex, ethnicity, race, and language spoken at home). The PREIS grantees have an average of 180 participants each per year. We estimate that it will take each grantee approximately 3 hours to enter the data and upload it.

Average burden hours to program participants

The only burden to participants is to respond to the perceived impact questions—for PREIS grantees only. Based on current data collection information, we estimate that it will take the 2000 PREIS participants approximately 5 minutes each to respond to the 10 questions.

Estimated annualized burden hours

Calculation of the total estimated annualized burden hours is shown in Exhibit 4.

As calculated above, we estimate that it will take each of the 106 grantees 4 hours to report their data related to reach and retention, 1 hour to report on measures of dissemination and dosage, and 2 hours to report on measures of fidelity, for a total of 7 hours each. Thus, it will take the 106 grantees a total of 742 hours to complete the form for these measures (see Appendix C).

For the participant-level measures (see Appendix D), the 45 Tier 1 C/D, Tier 2 and PREIS grantees will need 1 hour each, for a total of 45 hours.

For the perceived impact questions, we estimate that it will take the 2000 PREIS participants 5 minutes each to respond to the questions (see Appendix E) for a total burden of 167 hours. We estimate that it will take the 11 PREIS grantees 3 hours each to enter the data and report them, for a total of 33 hours.

The total burden to participants and grantees is 987 hours.

Exhibit 4. Estimated Annualized Burden Hours

Forms (If necessary)	Type of Respondent	Number of Respondents	Number of Responses per Respondent	Average Burden Hours per Response	Total Burden Hours
Measures for all grantees	Grantee program staff—all grantees	106	1	7	742
Participant-level measures	Grantee program staff—Tier 1 C/D, Tier 2, and PREIS grantees	45	1	1	45
Perceived impact questions	Youth participating in PREIS programs	2000	1	5/60	167
Perceived impact measures	Grantee program staff—PREIS grantees	11	1	3	33
Total		2,106			987

b. Estimated Annualized Cost to Respondents

The estimated 1-year annualized cost to respondent is shown in Exhibit 5. Salaries of the grantee staff collecting data, entering data, and summarizing and reporting data will vary widely. We estimate an average hourly rate of $30. Approximately 800 of the youth participating in the PREIS program are age 16 are older. We estimate that half (400) of them are working at an hourly rate of $8. The total burden hours for 400 youth to complete the 5-minute questionnaire would be 33 hours.

Exhibit 5. Estimated 1-Year Annualized Cost to Respondents

Forms (If necessary)	Type of Respondent	Number of Respondents	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
Performance measure reporting form	Grantee program staff—all grantees	106	742	$30.00	$22,260
Participant-level measures	Grantee program staff—Tier 1 C/D and Tier 2	45	45	$30.00	$1,350
Perceived impact questions	Youth participating in PREIS programs	400	33	$8.00	$264
Perceived impact measures	Grantee program staff—PREIS grantees	11	33	$30.00	$990
Total					$24,864

A.13 Capital Costs (Maintenance of Capital Costs)

There are no capital costs associated with this study.

A.14 Cost to Federal Government

With the expected extended period of performance, cost to the federal government is estimated to be $219,299. This is the cost estimated by the contractor, RTI International, and includes the estimated cost of coordination with OAH, RTI IRB and OMB applications, maintenance of the data reporting system, training and technical assistance to the grantees and OAH/FYSB staff in the use of the data reporting system, and data analysis and reporting.

A.15 Program or Burden Changes

The request is an adjustment that will decrease the annual burden from 3,428 hours to 987 hours. This decrease is primarily due to the fact that the PREIS grantees have a much lower number of program participants on average (180) than the 1000 per grantee that was previously estimated. In addition, there will be just a single data collection during the 7-month extension, rather than the two data collections that previously occurred in a full year.

A.16 Tabulation of Data and Schedule

The grantees will continue to collect data during their 5^th year of program implementation, and report the data at year’s end (September 30^th (TPP) and November 30^th (PREIS)). RTI will then analyze the data and prepare a written report, summarizing findings. Data will be broken down by type of grantee (i.e., Tier 1, Tier 2, or PREIS). Participant-level data will also be analyzed according to key characteristics (e.g., gender, race/ethnicity, and age).

A.17. Display of Expiration Date for OMB Approval

The OMB approval number and expiration date will be displayed on all data collection instruments.

A.18. Exceptions to Certification Statement.

There are no exceptions to the certification statement.

Appendix A

Federal Register Notice to the Public

Appendix B

RTI Institutional Review Board Notice

Appendix C

Performance Measures for Tier 1 A/B Grantees

Appendix D

Performance Measures

for Tier 1 C/D, Tier 2, and PREIS grantees

Appendix E

Perceived Impact Questions

(For youth in PREIS programs)

OMB No. 0990-0392

Exp. Date 5/31/2015

Date ______/_______/______

Demographic Questions

1. In what month and year were you born?

MARK (X) ONE Month and One Year

• January  2002

• February  2001

• March  2000

• April  1999

• May  1998

• June  1997

• July  1996

• August  1995

• September  1994

• October  1993

• November  1992

• December  1991

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-0392 . The time required to complete this information collection is estimated to average 5 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer.

2. What grade are you in? (If you are currently on vacation between grades, please indicate the grade you will be in when you go back to school).   

MARK (X) ONE ANSWER

• 6^th

• 7^th

• 8^th

• 9^th

• 10^th

• 11^th

• 12^th

• Ungraded

• College/Technical school

• Not currently in school

2. Are you male or female?

MARK (X) ONE ANSWER

• Male

• Female

2. Are you Hispanic or Latino?

MARK (X) ONE ANSWER

• Yes

• No

2. What is your race?

YOU MAY MARK (X) MORE THAN ONE ANSWER

• American Indian or Alaska Native

• Asian

• Black or African-American

• Native Hawaiian or Other Pacific Islander

• White

• Some other race (please specify):___________________________

2. When you are at home or with your family, what language or languages do you usually speak?

YOU MAY MARK (X) MORE THAN ONE ANSWER

• English

• Spanish

• Chinese language such as Mandarin or Cantonese

• Some other language: _________________________

Perceived Impact of Program

For the next few questions, please think about [NAME OF PROGRAM] and how it may have influenced you. You may not have thought about these situations before, but please still answer the questions. Think about what you would do and answer as best you can.

1. Would you say that being in [NAME OF PROGRAM] has made you more or less likely to have sexual intercourse in the next year?

• Much more likely

• More likely

• About the same

• Less likely

• Much less likely

2. Would you say that being in [NAME OF PROGRAM] has made you more or less likely to abstain from sexual intercourse in the next year?

• Much more likely

• More likely

• About the same

• Less likely

• Much less likely

3. If you were to have sexual intercourse in the next year, would you say that being in [NAME OF PROGRAM] has made you more or less likely to use any of these methods of birth control?

• Condoms

• Birth control pills

• The shot (Depo Provera)

• The patch

• The ring (NuvaRing)

• IUD (Mirena or Paragard)

• Implant (Implanon)

• Much more likely

• More likely

• About the same

• Less likely

• Much less likely

4. If you were to have sexual intercourse in the next year, would you say that being in [NAME OF PROGRAM] has made you more or less likely to use a condom?

• Much more likely

• More likely

• About the same

• Less likely

• Much less likely

Appendix F

Observation Form

(For all grantees)

Appendix G

Fidelity Process Report Form

(For all grantees)

1 Observation form is included in Appendix F

2 Fidelity process report from is included in Appendix G

53