Supporting Statement
For
Program Evaluation of the I Can Do It, You Can Do It Health Promotion Program for Children and Youth With Disabilities
New OMB Application
Project Officer: Eileen Elias, Deputy Director
Office on Disability
U.S. Department of Health & Human Services
200 Independence
Avenue, S.W., Room 637D
Washington, D.C. 20201
Email: [email protected]
Phone: (202) 401-5844
TTY: (202) 205-8280
Fax: (202) 260-3053
TABLE OF CONTENTS
A. Justification……………………………………………………………………………..1
Circumstances Making the Collection of Information Necessary…………………. 1
Purpose and Use of the Information Collection……………………………………. 3
Use of Improved Information Technology and Burden Reduction…………………5
Efforts to Identify Duplication and Use of Similar Information…………………… 6
Impact on Small Businesses or Other Small Entities………………………………. 6
Consequences of Collecting the Information Less Frequently…………………….. 6
Special Circumstances Relating to the Guidelines of 5 CFR 1320.5……………… 7
Comments in Response to the Federal Register Notice and Efforts to Consult
Outside the Agency ……………………………………………………………….. 7
Explanation of Any Payment or Gift to Respondents……………………………… 8
Assurance of Confidentiality Provided to Respondents…………………………… 8
Justification for Sensitive Questions………………………………………………. 9
Estimates of Annualized Burden Hours and Costs………………………………... 10
Capital Costs……………………………………………………………………….. 12
Annualized Cost to the Federal Government………………………………………. 12
Explanation for Program Changes or Adjustments………………………………... 13
Plans for Tabulation and Publication and Project Time Schedule………………… 13
Reason(s) Display of OMB Expiration Date is Inappropriate ……………………. 15
Exceptions to Certification for Paperwork Reduction Act Submissions………….. 15
B. Statistical Methods (used for collection of information employing statistical methods)
Respondent Universe and Sampling Methods………………………………………15
Procedures for the Collection of Information……………………………………….16
Methods to Maximize Response Rates and Deal with Nonresponse………………. 17
Test of Procedures or Methods to be Undertaken………………………………….. 17
Individuals Consulted on Statistical Aspects and Individuals Collecting
and/or Analyzing Data…………………………………………………………….. 17
TABLES
A-1: Estimated Annualized Burden Table………………………………………………… 11
A-2: Cost to Respondents…………………………………………………………………. 12
A-3: Annualized Federal Costs……………………………………………………………. 13
A-4: Project Time Schedule……………………………………………………………….. 14
Program Evaluation of the I Can Do It, You Can Do It Health Promotion Program for Children and Youth With Disabilities
A. JUSTIFICATION
The purpose of this submission is to request OMB approval to conduct a longitudinal evaluation of participants in a health promotion program developed by the Office on Disability (OD) targeted at children and youth with disabilities. Aggregate responses to the surveys administered to participating children and youth at three points in time (before the program begins, at the end of the program and eight months after the end of the program) will be used to evaluate the efficacy of the program and make adjustments as necessary to future offerings to increase its effectiveness.
1. Circumstances Making the Collection of Information Necessary
In 2005, the Surgeon General of the United States issued the Call to Action to Improve the Health and Wellness of Persons with Disabilities. The report documented that a significantly lower percentage of persons with disabilities than those without report their health to be excellent or very good (28.4% versus 61.4%) (Centers for Disease Control and Prevention, 2004). Goal three of the report was that "[p]ersons with disabilities can promote their own good health by developing and maintaining healthy lifestyles."
One important source of the Surgeon General's report was the U.S. Department of Health and Human Services’ (HHS) Healthy People 2010 initiative, which outlined a strategy for eliminating health disparities and improving the health of the nation. This initiative includes important health promotion and disease prevention goals written with people with and without disabilities. Objectives in three of the chapters of Healthy People 2010 target children and youth with and without disabilities:
Two objectives in Chapter 6, “Disability and Secondary Conditions” are:
6-2. Reduce the proportion of children and adolescents with disabilities who are reported to be sad, unhappy, or depressed.
6-13. Increase the number of Tribes, States, and the District of Columbia that have public health surveillance and health promotion programs for people with disabilities and caregivers.
Two objectives in Chapter 19, "Nutrition and Overweight," are:
19-3. Reduce the proportion of children and adolescents who are overweight or obese.
19-5. Increase the proportion of persons aged 2 years and older who consume at least two daily servings of fruit.
Finally, two objectives in Chapter 22, "Physical Activity and Fitness," are:
22-6 Increase the proportion of adolescents who engage in moderate physical activity for at least 30 minutes on 5 or more of the previous 7 days.
22-11. Increase the proportion of adolescents who view television 2 or fewer hours on a school day.
The OD oversees the implementation and coordination of disability programs, policies and special initiatives pertaining to the over 54 million persons with disabilities in the United States. As part of these efforts, the OD promotes the health of children and youth with disabilities. Of particular interest is how children and youth with a wide range of physical, sensory, developmental/cognitive, and/or behavioral health (emotional and/or substance abuse) disabilities can be encouraged to adopt a healthier life style that includes good nutrition and increased physical activity.
“I Can Do It, You Can Do It” is a health promotion intervention program developed by the OD targeted at children and youth between the ages of 10 and 21 with a wide range of physical, sensory, developmental/cognitive, and/or behavioral health (emotional and/or substance abuse) disabilities and a reading or comprehension level of 6th grade or above. The goals of the program are consistent with the objectives of the Surgeon General’s Call to Action and Healthy People 2010 described above: (a) change the behaviors of participants in two areas -- increase their level of physical activity and increase their positive nutrition habits -- and (b) increase their socio-emotional health.
The program links adults with and without disabilities with children and youth with disabilities (mentees) in a one-on-one eight-week mentoring program. Mentoring is an approach that has been well-documented in the research literature as efficacious in changing health behaviors of individuals with and without disabilities.
Up to 13 organizations will be recruited to serve as the intermediaries between the program evaluation and the mentor/participant pairs. These cooperating organizations -- public, non-profit or private organizations who work with children and youth with disabilities -- will be recruited to implement the program. Sponsoring organizations pair volunteer adult mentors with children and youth with disabilities (mentees) who wish to participate in the program.
One individual from each of the cooperating agencies will be designated as the “agency coordinator,” who will coordinate the implementation of the program at that organization, including:
Recruiting mentors;
Recruiting participants/mentees;
Ensuring that mentors receive appropriate technical assistance and advice during the eight week program; and
Maintaining lists of participants/mentees and their code numbers in order to ensure confidentiality of responses. (see section 10, "Assurance of Confidentiality Provided to Respondent.")
This study represents an opportunity to:
Document the effectiveness of the program;
Advance our understanding of how to encourage children and youth with disabilities to adopt and maintain a healthy lifestyle;
Advance our understanding of perceived barriers to maintaining a healthy lifestyle among this segment of the United States population; and
Assess the effectiveness of strategies to increase utilization of the program on a national basis.
The following sections of the U.S. Federal Code authorize the collection of information for this study: 42 USC 241, Section 301 of the Public Health Service Act, 42 USC 247b-4 and Section 317 C of the Public Health Service Act created by public law 106-310 (see Attachment 1).
2. Purpose and Use of the Information Collection
Three types of information will be collected.
An on-line survey, available in alternate formats as required, will be completed by participating children and youth/mentees.
An on-line process evaluation quantitative instrument will be completed by mentors at the end of the program.
An on-line process evaluation quantitative instrument will be completed by up to 13 agency coordinators at the end of the program.
Participant Survey
The purpose of the participant survey is to collect quantitative data from participants/mentees in the eight week program that will measure the extent to which the program has had an impact on several categories of outcome indicators aligned with the Surgeon General's Call to Action and Healthy People 2010, including
Physical activity behaviors;
Nutritional behaviors, including types of foods consumed;
Socio-emotional health; and
Utilization of health care.
“I Can Do It, You Can Do It” is one of the relatively few health promotion programs that has been adapted to incorporate the needs of people with physical, sensory, developmental/cognitive, and/or behavioral health (emotional and/or substance abuse) disabilities by including such factors as altered goals for physical activity (e.g., selection of physical activities and altered times for increased physical activity based on the individual's disabling condition). In addition, the individual will be educated about good nutritional practices and will be encouraged to apply these daily with the support of the parents.
The survey data from three points in time -- a pre-test before the program begins, a first post-test immediately at the conclusion of the eight week program, and a second post-test eight months after the conclusion of the program - will be analyzed to seek statistically significant differences before and immediately after the program, and to determine the longitudinal impact of the program (e.g., is there "slippage" in any observed impact after an eight month period?). The OD will use these results to: (a) obtain essential information about the numbers of children and youth who participate in the Program, the numbers of mentors and the ratio of mentors to mentees; grouped demographic information about these participants, and other measures essential for reporting the scope, magnitude and growth of the national program; (b) determine the overall efficacy of the program; and (c) examine specific sub goals, including the following:
Does the efficacy of “I Can Do It, You Can Do It” vary with the type of disability, including subpopulations with various types of physical,developmental/cognitive, sensory, and/or behavioral (emotional and/or substance abuse) disabilities?
Does the efficacy of “I Can Do It, You Can Do It” vary across sociodemographic characteristics, including gender, age, socio-economic status, ethnic group and geographic setting (urban vs. rural)?
Does the efficacy of “I Can Do It, You Can Do It” vary across one or more specific health outcomes (utilization of health care, general physical health, social and emotional health, or decreased incidence of secondary conditions) in relation to particular types of disability, gender, geographic settings, or sociodemographic characteristics?
The majority of items in the participant survey have been taken from existing normed, validated instruments including the Center for Disease Control’s (CDC) Behavioral Risk Factor Surveillance System (BRFSS) and the National Health and Nutrition Examination Survey (NHANES), both annual surveys conducted in all 50 states by the CDC through state health departments. Demographic questions align with the U.S. Census Bureau’s Survey of Income and Program Participation (SIPP).
Process Evaluation
In addition, a two-part process evaluation will take place at the end of the program. Information from the process evaluation will be used to determine what parts of the program were successful, the usefulness of program materials, and what changes should be made to improve the administration of the program.
A quantitative survey will be administered to mentors at the end of the program. Survey items target: (a) perceptions of mentors regarding what parts of the program worked well; (b) their assessment of program materials and technical assistance; (c) the types of activities they undertook with program participants; and (d) suggestions for improvements to specific parts of the program for future implementations.
A structured quantitative instrument will be completed by each agency coordinator and will focus on the following: (a) strategies they used to recruit mentors and mentees; (b) their assessment of program materials, including manuals, forms, and other resources; and (c) identification of challenges in the administration of the program and suggested improvements for possible future implementations.
Aggregate findings from the impact and process evaluations will be shared with other organizations in the public and non-profit sectors that are involved with promoting the health of people with disabilities, and will be used to advance our knowledge of strategies that are effective in fostering change in physical activity and nutrition behavior among children and youth with disabilities. The efficacy of the program or needed improvements could not be determined without collecting these data.
3. Use of Improved Information Technology and Burden Reduction
The use of on-line HTML-formatted surveys for the participant surveys, the mentor process evaluation instrument and the agency coordinator evaluation instrument will significantly reduce the burden on these respondents. The participant, mentor and agency coordinator surveys will allow respondents to use standard/universal keyboard and mouse commands to select responses to each question that appears on the screen. After finishing the survey, the respondent will click on a submit button, causing his or her answers to be electronically transmitted to a database.
The surveys will be programmed to follow designated skip patterns based on the respondent’s answers to previous questions. For example, a series of items on the participant survey asks about participation in organized school-based physical activity programs. The root question in this series asks whether their school has such a program. If the respondent answers "no," none of the follow-up items in that series will be seen by the respondent.
The surveys will be section 508-compliant, meaning that they will have graphics explained via pop-up text boxes and will be readable by screen reading programs such as Dragon Naturally Speaking.
In prior research, the use of on-line surveys has been found to increase response rates, particularly when token incentives are used when surveys are completed (see section 9, "Payment/Gift to Respondents" below).1 The use of web-based surveys also enables a significant number of improvements to both efficiency and effectiveness to be made in the collection of survey data compared to either paper-and-pencil methods or interviewer-assisted methods.
There is increased privacy for respondents, as well as control over when and where to complete the survey. Respondents will be able to save partially-completed surveys and return to them at a later time or date. Respondents will be able to enter their own responses directly into the web-based survey instead of having to tell an interviewer his or her answers to the questions. In addition, all web-based surveys will utilize secure socket layer (SSL) technology to ensure that data is encrypted during transmission.
Multiple costs associated with paper and pencil surveys, including printing costs and mailing costs for blank surveys out and completed surveys in are eliminated.
Use of the web-based surveys eliminates the need for hard copies of surveys and provides for secure, automatic back-up of data.
Error due to respondent error (e.g., the on-line surveys are coded to reject and supply error messages for such things as out-of-range responses) and data entry error are eliminated.
The software application to be used (Inquisite) automates the process of producing basic descriptive statistics (frequency counts and percentages) as well as automatic transfer to statistical applications.
The on-line surveys will collect only the minimum information necessary for the purposes of the project.
4. Efforts to Identify Duplication and Use of Similar Information
There is a small but growing body on the impact of mentoring-based health promotion programs on people with disabilities. For example, the health promotion intervention Living Well With a Disability, developed at the University of Montana, has a longitudinal database of responses to impact items stretching back over five years. However, The University of Montana developed intervention is substantially dissimilar to the health promotion intervention being measured and evaluated by the “I Can Do It, You Can Do It”. Impact data from other health promotion interventions cannot be used to assess the efficacy of the “I Can Do It, You Can Do It” program which is focused on applied increased physical activity and improved nutritional behaviors for children and youth with disabilities.
5. Impact on Small Businesses or Other Small Entities
No small businesses will be involved in this study.
6. Consequences of Collecting the Information Less Frequently
Participants/mentees will be asked to complete a registration form and a pre-test impact survey before the beginning of the program, a post-test of the same impact survey immediately at the end of the program and a second post-test eight months after the program has ended. Each survey is completed only once. The consequences of not collecting the impact information from participants/mentees include not being able to assess the impact of the program. A pre- and post- test design is needed to detect changes that occur due to the program.
The consequences of not collecting data on the second post-test would be severe, and include not knowing the lasting impact that the program may have beyond the time it ends. The issue of lasting effects of health promotion programs is one that has been recognized in the literature as significant, and one that has substantial health and fiscal implications (e.g., if participants utilize less health care because of their participation in the program, does the effect last over time?).
The consequences of not conducting the process evaluations with mentors and agency coordinators (on-line surveys) would be to not gain valuable information from those most actively involved in working with participants or coordinating the program on what parts of the program worked well, what activities they undertook (which can become suggestions for future administrations) and what parts of the program are in need of improvement.
Overall, the frequency of data collection is occasional.
7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
Only one of the special circumstances described in 5 CFR 1320.5 applies to the proposed collection of information – participants/mentees will be asked to complete the pre-test and post-test with an eight-week gap between them. This is necessary to capture data immediately before and just after the end of the program to measure the effect that the program has at its conclusion.
This information collection request complies with the other portions of 5 CFR 1320.5.
8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
A 60-Day Federal Register notice was published on January 2, 2008. No public comments were received.
To ensure that this data collection is not duplicative and the study design and instruments are appropriate, a literature review was completed and found that studies addressing the impact of mentoring-based health promotion programs for the targeted population of children and youth with disabilities are sparse. Therefore, this program evaluation is a logical step in determining the impact of this specific program.
No persons outside the HHS OD and contractors were consulted. The OD and New Editions (OD contractor) staff involved in the preparation of these documents were:
Margaret Giannini, M.D., Office on Disability
Eileen Elias, Office on Disability
Michael Marge, Ed.D., Office on Disability
Betsy Tewey, New Editions
Shelia Newman, New Editions
Anthony Cahill, Ph.D., University of New Mexico School of Medicine (consultant with New Editions)
Roberta Carlin, J.D., American Association on Health and Disability (consultant with New Editions)
9. Explanation of Any Payment or Gift to Respondents
Participants/mentees will be provided with low-cost material incentives when they complete the pre-test and first and second post-tests. Prior experimental research has demonstrated that offering small incentives increases the response rate in both web-based and paper-and-pencil surveys and that small, token incentives have results approximately equal to that of larger incentives .2 Additional research has found that the motivating power of incentives lies in terms of its symbolic, rather than monetary, value.3
Incentives will be chosen on the basis of their potential interest to children and youth, and will include such things as FM-Caribiner radios, disposable 35 mm cameras, notepads and other incentives. No single incentive will cost more than $3.00. Participants will be notified about the incentives by the agency coordinators in advance of each survey, and incentives will be sent to participants who complete each survey through the agency coordinator.
Sponsoring organizations and agencies will receive awards that range from $3000-$5000 to implement the program and participate in the evaluation. This includes the agency coordinator time to complete the process evaluation form.
10. Assurance of Confidentiality Provided to Respondents
Personal identifiers, including the name of the participant/mentee, contact information and contact information for parents or guardians are requested on the program registration form. Agency coordinators will use this information to maintain contact with participants/mentees and assign participants to mentors.
However, this information will not be transmitted to the contractor or any information that could personally identify a participant/mentee or link a specific participant to his or her responses will be seen by the contractor staff analyzing the data. The registration forms will be maintained at the participating agencies and will not be sent to the contractor staff. Each participant will be assigned a unique code number that will be used when participants complete the pre- and post- evaluation instruments. Aggregate data will be analyzed and reported out.
Because of the on-line survey, respondents will be able to complete the surveys at a location of their choice. Neither the staff of participating agencies nor contractor staff analyzing the survey will know whether any individual participant has completed the survey. Contractor staff will not have access to the list of participants. Respondents are urged in the web-site introduction not to answer questions they do not wish to, nor questions that make them feel uncomfortable.
Participant/mentee responses will be kept private to the extent allowed by law by keeping individual identifiers separate from the survey database. Staff of the participating agencies will keep a list of the individual associated with an individual code, but will not have access to completed surveys. Contractor staff will have access to linked codes across pre- and post-tests as well as the completed instruments, but will not have access to the code list which links an instrument to a particular individual name. Links will be destroyed once contract staff are insured that data are accurate and entered correctly, approximately three months after the last post-tests are due.
11. Justification for Sensitive Questions
Information that some participants/mentees may regard as sensitive will be collected on items six and seven of the participant registration form. These items ask respondents to identify their ethnic status and race, using HHS-approved items. These items are justified because it is important to determine whether the program has differential impacts on participants from different ethnic or racial groups. Should a significant differential impact be found based on ethnic status or race, it will be an important factor in examining program materials and approaches to identify potential bias.
Information that some participants may regard as sensitive will be collected on items 26, 27, 28 29, 30, and 31 and of the participant survey. Some respondents may feel embarrassed about providing answers to one or more of these items, which concern personal physical and socio-emotional health. All respondents will be given the opportunity to not answer items which may make them feel uncomfortable.
Item 26 asks respondents about the extent to which they are independent or in need of assistance with four activities of daily living: basic bathing/washing needs, basic dressing needs, basic toilet needs and taking medications.
Item 27 concerns limitations on activity and independence due to xx secondary conditions: contractures, circulatory problems, joint and muscle pain and sleep problems.
Item 28 asks respondents whether they have been treated by a mental health professional.
Item 29 asks respondents to assess the extent to which they are happy or unhappy with their lives.
Item 30 asks respondents to assess their socio-emotional health.
Item 31 asks respondents the extent to which they are satisfied with their friendships and social life.
These items are justified for two reasons. Research has demonstrated that these factors can be influenced by participation in health promotion programs targeted at people with disabilities.4 These items will assess the impact of this health promotion program on these factors.
12. Estimates of Annualized Burden Hours and Costs
A. Annualized Burden Hours Estimate
The following data collection forms will be used in the proposed project.
Before enrolling in the program, participants/mentees will complete a registration form containing socio-demographic and contact information.
At the beginning of the program, each participant/mentee will meet with their mentors to choose appropriate goals and types of activities using a one-page Goal Setting Worksheet.
At the beginning of the program, participants/mentees will complete a pre-test.
During the eight weeks, participants/mentees meet with their mentors one or more times each week to review how well the plan they designed is being conducted and make changes as needed, using a Weekly Check-In form that helps participants and mentors assess the extent to which goals are being met and to identify possible changes in their plans.
At the conclusion of the program, participants/mentees complete a first post-test using the same instrument as the pre-test.
At the conclusion of the program, mentors will be asked to complete an on-line quantitative survey that will be used as part of the process evaluation.
At the conclusion of the program, agency coordinators will be asked to complete and on-line quantitative survey that will be used as part of the process evaluation.
Eight months after the conclusion of the program, participants/mentees complete a second post-test using the same instrument as the pre-test and first post-test.
Estimates of burden are based on timed practice survey completions or interviews we completed as part of the instrument development process. The respondent burden is summarized below in Table A-1.
Table A-1
Estimated Annualized Burden Table
Forms
|
Type of Respondent
|
Number of Respondents |
Number of Responses per Respondent |
Average Burden Hours per Response |
Total Burden Hours |
Registration Form |
Program Participant/Mentee |
660 |
1 |
8/60 |
88 |
Goal Setting Worksheet |
Program Participant/Mentee |
610 |
1 |
7/60 |
71 |
Mentor Registration Form |
Mentor |
450 |
1 |
10/60 |
75 |
Pre-Test Survey |
Program Participant/Mentee |
560 |
1 |
19/60 |
177 |
Weekly Check-In Form |
Program Participant/Mentee |
560 |
8 |
7/60 |
522 |
First Post-Test Survey |
Program Participant/Mentee |
510 |
1 |
18/60 |
153 |
Second Post-Test Survey |
Program Participant/Mentee |
460 |
1 |
18/60 |
138 |
Mentor Post Assessment |
Mentor |
450 |
1 |
15/60 |
112 |
Agency Coordinator Survey |
Agency Coordinators |
6 |
1 |
45/60 |
4.5 |
Total |
|
|
|
|
1340.5 |
B. Annualized Cost to Respondents For Hour Burdens
Hourly wage rates for program participants, who are children and youth with disabilities, have been estimated at $0, since many or most of these individuals will not be in permanent employment.
Table A-2
Cost To Respondents
Type of Respondents |
Total Burden Hours |
Hourly Wage Rate |
Total Respondent Costs |
Program Participants |
1150 |
$0 |
$0 |
Mentors |
187 |
$50 |
$9,375 |
Agency Coordinators |
4.5 |
$50 |
$225 |
Total |
1341.5 |
|
$9,600 |
13. Capital Costs
There are no capital, start-up, operation or maintenance costs to respondents resulting from the collection of information.
14. Annualized Cost to the Federal Government
The contractor’s costs are based on estimates provided by the contractor who will carry out the data collection activities. The OD contracted with New Editions Consulting, Inc., to develop the surveys, collect the data, analyze the data, and produce a report. The OD will provide oversight of the contractor and project. Shelia Newman, President of New Editions, [(703) 356-8035] will oversee all data collection activities. Ms. Newman produced the cost estimates based on staffing requirements, wages, other direct costs (ODCs) and expected expenditures of similar projects. Current plans are to conduct this survey once, and costs are estimated for the entire costs of the administration of this survey.
The estimated Federal costs associated with conducting the “I Can Do It, You Can Do It” program evaluation, analyzing the data, and writing the final report amount to $103,421. These costs are summarized in Table A-3 below.
Table A-3
Annualized Federal Costs
RFP Process (contractor) |
Services of a Project Director and administrative support staff to recruit and select cooperating agencies, make awards. 50 hrs @ $100/hr rate |
$5,000 |
Participating Site Awards (contractor) |
To be distributed among 8-13 qualified sites, in amounts between $3000-$5000. |
$24,500 |
PRA Process (contractor) |
Services of Project Director and Evaluator to coordinate and prepare package, including forms, IRB application, and respond to questions. 25 hrs @ $125/hr rate
|
$3,125 |
Data Collection (contractor) |
Services of an Evaluator, Trainer and Research Assistant to revise forms, collect and maintain data, monitor data collection progress, provide training and technical assistance to participating sites on the data collection process and ODCs for computer expenses. 375 hrs @ $125/hr rate
|
46,875 |
Data Analysis and Report Preparation (contractor) |
Services of an Evaluator and Research Assistant to analyze data and prepare draft and final reports. 100 hrs @ $125/hr rate
|
$12,500 |
OD staff time: |
10% FTE for GS-15 for oversight |
$11,421 |
Total |
|
$103,421 |
15. Explanation for Program Changes or Adjustments
This is a new data collection.
16. Plans for Tabulation and Publication and Project Time Schedule
A. Analytic Plan
The survey data from the pre-test, first post-test and second post-test will be analyzed to evaluate the impact that the program has had on participants/mentees on the specified variables. Descriptive analyses to describe socio-demographic characteristics of respondents broken out by participating site and as whole will be conducted. Means and standard deviations and/or medians and interquartile ranges for continuous variables and frequencies for categorical variables will be computed. To determine changes over time on impact variables, standard analytical procedures including paired-sample, two tailed t-tests on means or Mann-Whitney tests for medians will be utilized. To identify changes based on socio-demographic characteristics (e.g., gender, ethnic identity, etc.) appropriate analytic techniques such as chi-square tests of independence will be used.
Results from the quantitative process evaluation of mentors will be analyzed using descriptive analyses, including means and standard deviations and/or medians and interquartile ranges for continuous variables and frequencies for categorical variables. For open-ended responses such as recommendations for improvements, a coding scheme to synthesize the data and reveal common response themes will be created.
B. Dissemination of Results
A summary report of the survey results will be prepared. The results of the program evaluation will be disseminated in several ways, including presentations at relevant professional association conferences. A copy of the final report will be posted on the OD web site (www.hhs.gov/od). All dissemination products will present aggregate results only; no individual responses that could identify a specific participant will be presented.
C. Timeline
Project Time Schedule
Title |
Activity |
Time Schedule |
Clearance Process |
Submit to HHS OMB liaison |
January, 2008 |
Clearance Approval |
Submit to OMB for approval |
May, 2008 |
Agency Recruitment |
Finalize agreements with cooperating agencies |
Upon receiving OMB clearance |
Mentor and Participant Recruitment |
Conduct agency and mentor orientations; recruit mentors and participants |
1-3 months after receiving OMB clearance |
Data collection |
Administer Web-based survey at three points in time |
First Wave: 3-6 months after OMB clearance for participant pre-test (start of program); 7-10 months after OMB clearance for first post-test (at conclusion of eight week program) and 15-19 months after OMB clearance for second post-test (eight months after conclusion of program)
For mentor process evaluation survey: 7-10 months after OMB clearance (at conclusion of program)
For agency coordinator process evaluation: 7-10 months after OMB clearance (at conclusion of program)
Second Wave: 6-9 months after OMB clearance for participant pre-test (start of program); 10-13 months after OMB clearance for first post-test (at conclusion of eight week program) and 18-22 months after OMB clearance for second post-test (eight months after conclusion of program)
For mentor process evaluation survey: 10-13 months after OMB clearance (at conclusion of program)
For agency coordinator process evaluation interviews: 10-13 months after OMB clearance (at conclusion of program)
Third Wave: 9-12 months after OMB clearance for participant pre-test (start of program); 13-16 months after OMB clearance for first post-test (at conclusion of eight week program) and 21-25 months after OMB clearance for second post-test (eight months after conclusion of program)
For mentor process evaluation survey: 13-16 months after OMB clearance (at conclusion of program)
For agency coordinator process evaluation interviews: 13-16 months after OMB clearance (at conclusion of program) |
|
|
|
Data analysis |
Produce statistics |
26-28 months after OMB clearance |
Draft Report |
Produce draft report |
29-31 months after OMB clearance |
Final Report |
Produce final report |
32-33 months after OMB clearance |
17. Reason(s) Display of OMB Expiration Date is Inappropriate
N/A
18. Exceptions to Certification for Paperwork Reduction Act Submissions
There are no exceptions to the certification.
B. STATISTICAL METHODS
1. Respondent Universe and Sampling Methods
The sample will be a non-representative sample of children and youth with physical,developmental/cognitive, sensory, and/or behavioral (emotional and/or substance abuse) disabilities. Results from the longitudinal pre-post study will not be used to generalize to the universe of children and youth with disabilities in the United States. Therefore, sample size and power are of secondary importance. No prediction that necessitates a requisite amount of power is being made.
2. Procedures for the Collection of Information
Recruitment of participants/mentees will be a two-stage process. In the first stage, cooperating agencies that agree to implement the program will be recruited. Potential cooperating agencies will be informed of several criteria that they must meet in order to participate, including:
Being able to recruit the required number of participants/mentees (120);
Being able to recruit sufficient numbers of mentors; and
Agreeing to not have participants enrolled in any other health promotion intervention during the project period.
Once cooperating agencies have been identified, agency coordinators will be responsible for recruiting participants.
A. Statistical Methodology For Stratification And Sample Selection
The sample will be a convenience sample. Stratification will not be used.
B. Estimation Procedure
The estimated number of respondents completing the registration form is 1,320. Attrition throughout the program is expected, with an estimated 1120 completing the pre-test survey, 1020 completing the post-test survey and 920 completing the second post-test survey. The estimated number of mentors completing the registration form and post assessment is 900. The estimated number of agency coordinators completing the survey is 13.
C. Degree of Accuracy Needed
The web-based survey was designed to meet the non-inferential goals of this study. The use of web-based surveys was selected to increase the accuracy of the data by: (a) providing increased privacy conducive to accurate and honest reporting by respondents, (b) eliminating the need for hardcopy records, (c) providing easy back-up to ensure no loss of data, and (d) reducing respondent error related to skip patterns.
Because the data will be entered directly by respondents, the only data entry errors will be on the part of the participants. Edit checks will be built into items as needed to ensure that respondents cannot give out-of-range or other inappropriate responses. Participants will be given identification numbers to enter in order to gain access to the survey, to ensure that participants do not complete the survey more than once.
D. Unusual Problems Requiring Specialized Sampling Procedures:
No unusual problems requiring specialized sampling procedures are anticipated.
E. Any use of less frequent than annual data collection to reduce burden
All questions have been limited to those considered essential to evaluate the efficacy of the program. This study is an ad hoc data collection (i.e., a one-time study), and therefore the data are collected less frequently than annually.
3. Methods to Maximize Response Rates and Deal with Nonresponse
The OD’s ability to gain the cooperation of agency coordinators is the key to the success of this endeavor. Through an orientation program for them, the use of small non-monetary incentives for participants/mentees, and the encouragement of mentors, the OD believes that the response rate will be high. Agency coordinators will be provided with the code numbers of participants who have not completed one of the three surveys, so they can be contacted and urged to complete the study. These contacts will be within the guidelines for the protection of human subjects to avoid even the appearance of coercion of respondents. Respondents will always be reminded that they are able to not respond to one or more items or an entire survey if they choose. Contacts with agency coordinators will be structured to simply remind them that they have not responded, and there is still time to do so.
4. Test of Procedures or Methods to be Undertaken
The OD plans to thoroughly test the web-based survey with a group of nine or fewer children and youth with disabilities prior to data collection. The complete study protocol, including the security of the transmission of data, the receipt system and other aspects of the program will be tested internally before the start of data collection.
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
JoAnn Thierry, Ph.D.
National Center on Birth Defects and Developmental Disabilities (NCBDDD)
Centers for Disease Control and Prevention
1600 Clifton Road, NE, MS: E-88
Atlanta, GA 30333
404-498-3022
1 Couper, M. (2000) Web Surveys: a review of issues and approaches. Public Opinion Quarterly 64, 464–494; Dillman, D. A. (2000) Mail and internet surveys: the tailored design method. J. Wiley, New York.; Schmidt, W. C. (1997) World-Wide Web Survey research: benefits, potential problems and solutions. Behaviour Research Methods, Instruments and Computers 29, 274–279
3 Shaw MJ, Beebe TJ, Adlis SA, Jensen H. The use of monetary incentives in a community survey: impact on response rates, data quality, and cost. Abstr Book Assoc Health Serv Res Meet. 1998; 15: 295-6; HealthSystem Minnesota, Institute for Research and Education, Health Research Center, Minneapolis, MN 55416, USA.
4 Gold, M., Siegel, J., Russell, L., & Weinstein, M. (1996). Cost-effectiveness in health and medicine. New York: Oxford University Press, Inc.; Marge, M. (1988). Health promotion for persons with disabilities: Moving beyond rehabilitation. American Journal of Health Promotion, 2, 29-44; Ravesloot, C., Young, Q.-R., Norris, K., Szalda-Petree, A., Seekins, T., White, G.W., Lopez, J.C., & Golden, K. Living well with a disability: A workbook for promoting health and wellness. 1994. Missoula, RTC; Seekins, T., White, G.W., Ravesloot, C., Norris, K., Szalda-Petree, A., Lopez, J.C., Golden, K., & Young, Q-R. (1999). Developing and evaluating community-based health promotion programs for people with disabilities; In R.J. Simeonsson & L.N. McDevitt (Eds.), Issues in disability & health: The role of secondary conditions & quality of life. (pp.221-238). Chapel Hill, NC: University of North Carolina, FPG Child Development Center
| File Type | application/msword |
| File Title | Supporting Statement |
| Author | Michael Marge |
| Last Modified By | DHHS |
| File Modified | 2008-05-07 |
| File Created | 2008-05-07 |