Supporting Statement for the
Paperwork Reduction Act of 1995
Part B. Collections of
Information Employing Statistical Methods
Demonstration and Evaluation of Community College Interventions for Youth and Young Adults with Disabilities
December 22, 2016
U.S. Department of Labor
200 Constitution Ave., NW
Washington, DC 20210
Table of Contents
COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
B.1 Respondent Universes and Sampling or Other Selection Methods 1
B.2 Procedures for Collection of Information 2
B.3 Methods for Maximizing Response Rates and Addressing Nonresponse 6
B.4 Tests of Procedures and Methods 8
B.5 Individuals Contacted on Statistical Aspects and/or Analyzing Data 9
PART B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
The U.S. Department of Labor (DOL) is requesting clearance for information collection to conduct (1) in-depth interviews with grantee staff, other community college administrators and staff, students, and grantee partner organizations, (2) focus groups with faculty, and (3) surveys of community college students. These data collections are essential elements of the evaluation of the Pathways to Careers: Community Colleges for Youth and Young Adults with Disabilities Demonstration Project.
Respondent Universes and Sampling or Other Selection Methods
Respondent universes
The data collection for the evaluation of the Pathways to Careers: Community Colleges for Youth and Young Adults with Disabilities Demonstration Project (PTC) includes several elements, including longitudinal survey of PTC participants; cross-sectional interviews of PTC participants; longitudinal interviews of PTC participants; focus groups with PTC program faculty; and in-depth interviews with community college administrators, office personnel, counselors, and PTC partner organizations.
PTC Participants—longitudinal survey
The PTC grantees will provide community college information about the program and its participants for the evaluation. However, some information will need to come directly from the participants by means of a self-report longitudinal survey, specifically, information on their program experience and on education and work outcomes.
The respondent universe for the survey is the total of all PTC program participants at Onondaga Community College (OCC) and Pellissippi State Community College (PSCC) who entered the program in academic years 2015-16 and 2016-17. Both grantees will enroll four cohorts of student participants: Cohort 1 enrolled in fall 2015, Cohort 2 enrolled in spring 2016, Cohort 3 enrolled in fall 2016, and Cohort 4 enrolled in spring 2017. Table B-1 presents estimated numbers for the PTC participant universe. OCC enrolled 38 participants and PSCC enrolled 74 participants in fall 2015 (and more participants in the spring and fall of 2016 as indicated in the table). The number of participants starting in spring 2017 at each college is an estimate. The total number of participants is expected to be 213.
Cohort |
Number of participants in OPC |
Number of participants in UPEP |
Total number of participants |
Aug-15 |
38 |
74 |
112 |
Jan-16 |
6 |
4 |
10 |
Aug-16 |
38 |
38 |
76 |
Jan-17 |
10 |
5 |
15 |
All |
92 |
121 |
213 |
PTC Participants—cross-sectional and longitudinal interviews
PTC participants will also be invited to participate in one-on-one interviews to capture information that cannot be collected through surveys. The following topics will be addressed through the interviews: the enrollment process; family engagement; school support and class experience; accommodations; interactions with faculty and other students; self-exploration and career exploration; individualized career planning; career preparation; and challenges.
PTC participants will be invited to participate in cross-sectional or longitudinal in-depth qualitative interviews. A cross-section of PTC participants from each of first year participants (cohorts 1 and 2) and from the second year participants (cohorts 3 and 4) at each of the grantee colleges in years 2, 3 and 4 of the evaluation will be invited. Similarly, first-year PTC participants and second-year PTC participants at each of the grantee colleges will be invited for longitudinal interviews (to see how their experiences change over time). Table B-2 presents the estimated number of PTC participants to be interviewed by type of interview and year as well as by college and cohorts. A total of 45 participants will complete the cross-sectional interviews and a separate group of 28 participants will complete the longitudinal interviews.
College |
Cohort |
Cross-sectional sample |
Longitudinal sample* |
TOTAL |
||||||||||
Spring 2016 |
Spring 2017 |
Fall 2017 |
Spring 2018 |
Fall 2018 |
Total |
Spring 2016 |
Spring 2017 |
Fall 2017 |
Spring 2018 |
Fall 2018 |
Total |
|||
OCC |
C1 & C2 |
3 |
5 |
-- |
5 |
-- |
13 |
4 |
3 |
2 |
-- |
-- |
9 |
22 |
C3 & C4 |
|
-- |
5 |
-- |
5 |
10 |
|
10 |
9 |
8 |
7 |
34 |
44 |
|
PSCC |
C1 & C2 |
2 |
5 |
-- |
5 |
-- |
12 |
4 |
3 |
2 |
-- |
-- |
9 |
21 |
C3 & C4 |
|
-- |
5 |
-- |
5 |
10 |
|
10 |
9 |
8 |
7 |
34 |
44 |
|
TOTAL |
5 |
10 |
10 |
10 |
10 |
45 |
8 |
26 |
22 |
15 |
15 |
86 |
131 |
|
* Estimated number of completed interviews assumes attrition of one participant per year for each sample.
PTC Program Faculty—focus groups
Three sets of focus group sessions with PTC program faculty are planned at each of the two colleges. The focus group will discuss experiences and perspectives related to involvement in PTC, responsibility for overseeing implementation of UDL, conducting professional development, and instructing the courses in which PTC participants are enrolled. Each focus group will consist of approximately eight faculty members. There are approximately 140 faculty members at PSCC that teach PTC core courses related to the 29 PTC career paths. At OCC, the number of faculty is much less because there are only five career paths. Assuming five faculty members per career path, we estimate that OCC would have, on average, about 25 faculty members that teach PTC core courses related to the five PTC career paths.
Community College Administrators, Office Personnel and PTC Partners—Interviews
The overarching design element of the site visits and qualitative data collection will be to obtain multiple perspectives on the course and extent of PTC implementation at the colleges. Each perspective will help illuminate the strategies, challenges, fidelity, and implications of local PTC program implementation. Determining the universes for the various community college personnel and partners is limited by the fact that the grant programs are evolving over the course of the grant period, with the hiring of new staff and bringing in new partners. The estimated universes for both colleges combined include: four college administrators; 40 partner organizations and 30 college personnel.
Selection Methods
Because of the expected small size of the PTC participant universe, we plan to invite all participants to participate in the longitudinal survey every 6 months, starting spring 2017 through the fall of 2018. No stratification or sampling will be used for the survey.
For the interviews, PTC participants will be contacted by mail and email. Through data use agreements, the two participating colleges will provide the evaluation contractor with a data file of contact information for each PTC participant. The PTC project directors will also be asked to send an advance letter/email to participants to introduce the study and contractor, and to encourage their participation. The contractor will use purposive selection to capture variation across specific participant characteristics, such as gender, prior college experience, and career path.
Purposive selection will be used for the college administrators, office personnel and PTC partners. All of the grantee administrators will be selected. The grantee administrators will be asked to identify the appropriate office personnel with knowledge and experience with the PTC program, including counselors and staff of related offices (e.g., Disability Services). The administrators will also be asked to identify their partner organizations, specifically those that are most involved in the program.
The grantee administrators will be asked to identify the PTC faculty members, who will then be recruited for annual focus group sessions through email and/or telephone contact. The aim is to have eight faculty members participate in a focus group session. We will use purposive selection to capture variation across faculty characteristics, such as faculty type (e.g., adjunct or other) and course subject.
Sampling unit, response rates, target sample sizes, and fielded sample sizes
Because the universe of PTC participants will be surveyed, no sampling is required. Table B-3 provides an estimate for the number of participants per cohort and the number that are expected to respond to the survey, assuming a 90 percent response rate at each wave. At the end of all waves of the survey, the overall response rate is estimated to be 65.7 percent.
Cohort |
Total number of participants |
Survey Dates |
Final Response rate |
|||
Spring 2017 |
Fall 2017 |
Spring 2018 |
Fall 2018 |
|||
Aug-15 |
112 |
101 |
91 |
82 |
74 |
66.1% |
Jan-16 |
10 |
9 |
8 |
7 |
6 |
60.0% |
Aug-16 |
76 |
68 |
61 |
55 |
49 |
64.5% |
Jan-17 |
15 |
-- |
13 |
12 |
11 |
73.3% |
All |
213 |
178 |
173 |
156 |
140 |
65.7% |
*Estimated number of completed interviews assumes 90% response at each wave.
For the PTC participant interviews, we estimate that that there will be 100 percent response for the cross-sectional interviews. For the longitudinal interviews, we estimate one participant per year will be lost through attrition and so we limit the number of interview waves for the first two cohorts. The response rate is estimated to be 50 percent at the third waves of the longitudinal interviews for cohorts 1 and 2. For cohorts 3 and 4, starting with 10 participants at each college, the response rate is expected to be 90 percent at the second wave and 80 percent at the third wave.
We expect 100 percent response for the in-depth interviews with college administrators and college personnel given the PTC program is part of their work. For partner organizations, 100 percent response is also expected because the numbers are few and the organizations are directly involved with the PTC program. Similarly, we expect 100 percent response for the faculty focus groups. Once the faculty members agree to participate, we expect all will attend the focus group sessions.
2. Procedures for Collection of Information
2.1. Statistical methodology for stratification and sample selection
No statistical methods are needed for stratification and sample selection for the participant survey. The universe of PTC participants will be invited to participate in the longitudinal survey.
No statistical methods are needed for stratification and sample selection for the qualitative interviews. As stated above, purposive selection will be used for the interviews with participants, college administrators and college personnel and with PTC program partner organizations.
No statistical methods are needed for stratification and sample selection for the faculty focus groups because purposive selection will be used. Following the invitations to faculty members at each college, the faculty that apply to participate will be reviewed and selected to provide variation by type of faculty (e.g., adjunct or other) and the type of courses taught.
2.2. Estimation procedures, variance estimation, for the surveys
Because the survey is a census, no estimation procedures are needed.
The survey is intended to provide descriptive information about the students that entered the PTC program in the two colleges, not to support comparisons between the participants and other students. So, the analysis will be descriptive, based on totals, means, and proportions on key variables of interest (e.g., student outcomes).
Because no sampling is used for the qualitative interviews, no estimation procedures are needed.
2.3. Degree of accuracy needed
Not applicable.
2.4 Unusual problems requiring special sampling procedures
None.
2.5 Use of periodic (less frequent than annual) data collection cycles to reduce burden
There is no use of less than annual data collection cycles because of the relatively short duration of the PTC grant program being evaluated.
3. Methods for Maximizing Response Rates and Addressing Nonresponse
3.1 Methods to maximize response rates
It is becoming more difficult to conduct telephone surveys as people move from traditional landline telephones and are less willing to accept calls on their cell phones. There is increasing evidence that web and paper questionnaires, using accepted contact procedures (e.g., followup mailings; Dillman, et al., 2009) can achieve response rates as high (or higher) than a telephone survey.
A web questionnaire, by itself, will not provide the coverage that a telephone survey will provide. Consequently, it is imperative for the PTC participant survey to use a multi-modal data collection approach, combining web administration with telephone followup of web nonrespondents to maximize response rates and minimize respondent burden.
Literature on survey research suggests that response rates in fact decrease when respondents are presented with multiple administration options at the same time. Instead, a more effective strategy is to offer respondents one survey mode at a time, and then implement a different mode with non-responders. For the PTC participant survey, survey participants will be first asked to complete the survey online via the Internet. The initial survey invitations to complete the survey online will be mailed and emailed to PTC participants. A weekly email reminder for completing the online survey will be sent to non-responders at the beginning of the second week of the data collection field period.
Three weeks after sending out the initial survey invitation, trained interviewers will begin calling those who did not respond to the web survey. A similar data collection and contact protocol will be used for each semi-annual follow-up survey.
Because of the longitudinal design of the survey, it is important to maintain accurate contact information and keep participants engaged throughout the study period to gain their cooperation for completing the semi-annual surveys. To achieve a high retention rate and minimize attrition, we propose to maintain contact with participants between data collection activities. Approximately three months after completing an initial or follow-up survey, we will email a note to all participants thanking them for their participation and reminding them to provide us with any updated contact information using the toll-free study number or the study-designated email address.
An incentive payment of $25.00 will also be provided to respondents to help maximize response and retention rates. Immediately upon receipt of a completed survey, we will mail a gift card for $25.00 to the participant’s preferred address. Furthermore, because web surveys are more cost efficient than telephone administration, we will offer participants an “early bird” bonus of an extra $5.00 if they complete the survey within three weeks of receiving the initial invitation (before we initiate the telephone administration).
Even though intensive methods will be used to increase response rates and convert non-responders, non-response bias is still a concern. High participation rates are not sufficient to rule out possible biases due to non-response. Therefore, response rates will be examined and if needed, we will adjust survey weights to account for non-response, as described in Section 3.2.
3.2 Weighting procedures for the surveys
Due to the small number of program participants, weighting will not be used. Weighting with such a small sample would likely give misleading results. The analysis will report counts, proportions, and measures of central tendency regarding program participants.
3.3 Methods to handle issues of non-response
Due to the small number of program participants and the focus of the analysis on documenting participant outcomes, no adjustments will be made for non-response.
4. Tests of Procedures or Methods
4.1. Pretesting and cognitive interviewing for the survey
Cognitive interviewing is planned to occur in April 2016 at each of the two grantee community colleges with up to four PTC participants per college. The interviews will focus on specific survey questions to ensure clarity, understanding and ability to respond appropriately. Results of the survey will be used to refine the survey instrument. No further testing of procedures or methods will be undertaken for completion of this survey.
4.2 Pretesting of the in-depth interviews
No testing of procedures or methods will be undertaken for completion of the first set of in-depth interviews. However, in the future, we will refine the interview and focus group moderator guides based our experience with the first administration to minimize burden and improve utility wherever possible.
5. Individuals Contacted on Statistical Aspects and/or Analyzing Data
The following individuals (see Table B-4) consulted on statistical aspects of the design and will also be primarily responsible for actually collecting and analyzing the data for the agency.
Table B-4. Persons and/or entities involved in data collection and/or analysis of data
Name |
Agency/Company/Organization |
Telephone Number |
Frank Bennici |
Senior Study Director, Westat |
301-738-3608 |
William Frey |
Vice-President, Westat |
301-610-5198 |
Doug Klayman |
President, Social Dynamics |
301-990-1105 |
Keith MacAllum |
Senior Study Director, Westat |
301-610-8854 |
Jarnee Riley |
Senior Study Director, Westat |
240-453-2724 |
Gary Shaheen |
Director of Policy & Programs, Social Dynamics |
301-990-1105 |
Richard S. Sigman |
Senior Statistician, Westat |
240-453-2783 |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Frank Bennici |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |