Study of Pell Grant Recipients Who Transfer Among Eligible Institutions
Request for OMB Clearance
Supporting Statement B
August 19, 2008
Contents
Section B. Methodology. 1
B1. Sample Design and Estimation. 1
B2. Survey Procedures. 2
B3. Methods to Maximize Response Rates. 5
B4. Tests of Procedures. 6
B5. Individuals Responsible for Study Design and Performance 6
Appendix A Description of Survey Items 7
Appendix B Letter to Pell Grant Recipient 11
Appendix C Westat Pledge of Confidentiality 12
List of tables
Table Page
Table 1 Standard errors for percentage of students who could not transfer all academic courses 4
Table 2 Standard errors for percentage of students with costs over some threshold 4
Table 3 Power for comparisons between small and large institutions 4
Table 4 Power for comparisons between racial/ethnic groups 5
Westat will select a sample of 668 students from the Pell Grant recipient file, provided by ED, which contains all Pell grantee recipients who have transferred from 2-year institutions to another institution in the 2005-06 school year. We will select 167 students each from the following categories of receiving institutions: 2-year public, 2-year private, 4-year public, and 4-year private.
Some variables will be used to form student strata, while other variables will be used to sort students within each stratum. A systematic sample will be selected within each stratum from the sorted listing of students. The allocation of sample across strata will be proportional to the number of students in each stratum. Stratification ensures that the proportion of the sample in categories of variables that are used to form the strata will be close to the proportion of all students in the categories. To illustrate the methodology for sampling with strata, suppose a stratum consists of 50 students and we wish to sample at a rate of 1 in 10. A random number between 1 and 10 is first selected. Suppose the random number is 6. The sample would then consist of the 6th, 16th, 26th, 36th, and 46th students on the sorted list. If the primary sort variable within the stratum was the geographic region, then this methodology ensures that the proportion of the sample from each region in the stratum is approximately equal to the proportion of students in the stratum that is from the region. Both the formation of strata and systematic sampling from sorted students within strata reduces sampling error and ensures representativeness for the selected variables. The Westat proprietary sampling software, WESSAMP, will be used to select the sample.
We wish to obtain an interviewed sample size of 200 students. To account for an expected 70 percent nonresponse rate, we plan to select 668 students. If the field period were longer, it would be possible to select substitutes for each sampled student who would only be interviewed when an initially sampled student was determined to be a nonrespondent.
The survey will be conducted primarily by telephone, with initial notification of the survey by mail and with the option of instead responding through the web. Westat’s strategy is to call respondents by phone within 1 week of the expected arrival of the introductory letter. We will then attempt to collect the responses by telephone, using CATI. The cover letter also will offer the option of responding through the web.
The transcript component of the study will be conducted by collecting transcripts either from students (who download and email the transcripts, or who fax printed copies of the transcripts) or from the sending and receiving institutions (for those students who give permission to collect transcripts but do not directly send the transcripts to Westat). Westat will reimburse institutions at their standard rate for sending the transcripts, and will reimburse students $10 for sending the transcripts directly. College catalogs will be obtained online to facilitate coding of the transcripts.
This study is intended to be a preliminary study, and will not produce national estimates. Therefore the data will not be weighted.
The Pell Grant recipients are unequally distributed, with many of the recipients located at a relatively small number of very large institutions in 2005-06. If all students are given the same probability of selection, then we will sample multiple students per institution from the larger institutions and relatively few from the smaller institutions. In some ways this is beneficial. For general statistics, such as the general types of courses that were accepted for credit, the sampling error will be minimized if students are sampled with equal probability. On the other hand, if one is interested in looking at particular classes of institutions, such as those institutions that have fewer than 1,000 students, then the sample size will be very small for such a class. Thus, if this is important, it may be desirable to sample students transferring to small institutions at a higher rate. For example, it could be that it is the smaller colleges that are more restrictive in accepting transfer credits. If we are looking for students who had problems (in order to understand how/why they had problems), then oversampling the small institutions would be useful.
We thus face two competing priorities. For some types of analysis, sampling error will be lower if small institutions are oversampled, but for other types of analysis such oversampling will increase sampling error. For ED, the highest priority is to have accurate overall estimates, and developing separate estimates for small institutions is not a key priority. Therefore, we will not oversample small institutions.
There are several variables for use in stratifying and sorting students prior to sample selection. The three variables that we consider to be most important will be used to form strata. These variables are:
Institution control (public, private non-profit, private profit-making)
Institution type
Enrollment size in the institution being transferred to
We expect that institutional control will be collapsed to two categories (public/private) for at least some strata, and that institution type will also be collapsed to fewer categories for at least some strata. For the enrollment size in the institution, there will be six categories with about equal numbers of cases in the central four categories, while the tail categories would likely have fewer than the others. This will depend on the distribution of this variable.
A number of strata will be determined, each of which is homogeneous with respect to these three variables. There will be no more than 72 strata. Some of these might be small and these would then be collapsed. For example, the 2-year transfer institution type might not be further stratified by the enrollment categories. As another example, the two private categories could be combined into one private category.
Within each stratum students will be sorted by the following variables, listed in priority order:
National Assessment of Educational Progress (NAEP) region
Student year
Institution
ID number
To help maximize the response rate, even given a short data collection time period, we will perform the following. (1) All sampled Pell Grant recipients will be sent a cover letter using ED stationery that describes the purpose of the survey. (2) The address and telephone number will be taken from the most recently available Pell Grant files, in order to ensure accuracy, and updated through a national change of address service. (3) The sample members will be contacted by telephone, helping to quickly confirm that the locating information is correct and allow tracing to begin if it is not. (4) Survey responses will be collected over the telephone, making maximum use of the telephone contact and avoiding the possibility of delay or of the subject forgetting to follow through. (5) Respondents will be given the option of responding through the Web or of calling a toll-free number if they wish to respond immediately. (6) The response status of each sampled Pell Grant will be monitored through an automated receipt control system, so that failures to reach a subject or to get a response can be followed up quickly. (7) The study members will be offered a financial incentive of $10 that will be included with the original letter notifying the student of the survey. (8) To facilitate collection of transcripts, respondents will be asked to sign and mail permission forms to Westat; however, we will also collect data on the telephone and web surveys on whether students gave permission to collect their transcript so that we can tell institutions that we have permission even if we do not received the signed permission forms. We will pay students $10 to send both transcripts directly to Westat in lieu of the reduced data collection costs for Westat.
All questionnaires that are received will be reviewed for consistency and completeness; if a questionnaire has too few items completed to be counted as a response (or if it has missing or conflicting data on key items), telephone interviewers will seek to obtain more complete responses. Interviewers who have received training in telephone interview techniques and specific training on the survey will conduct all of the telephone interviews.
The questionnaire has been reviewed extensively by both Westat and ED. Westat will perform a test of the computer software for both the CATI and the web version of the questionnaire in order to verify that respondents progress appropriately through the questionnaire, and that data are accurately recorded.
This study itself is a test of the procedures needed to accurately measure students’ experiences when transferring from one institution to another: e.g., whether students are able to report on their experiences accurately, or whether transcripts are needed in order to have accurate information.
Several key staff are responsible for the study design and performance. These individuals are as follows:
Archie Cubarrubia, Project Officer, Office of the Under Secretary, U.S. Department of Education (202-260-5548)
Vickie Schray, Office of the Under Secretary, U.S. Department of Education
(202-401-1021)
David Bergeron, Office of Postsecondary Education (202-502-7815)
Dennis Berry, Office of Vocational and Adult Education (202-245-7814)
Greg Henschel, Office of Vocational and Adult Education (202-245-7661)
Tom Weko, National Center for Education Statistics (202-502-7643)
Michelle Coon, National Center for Education Statistics (202-502-7357)
Brad Chaney, Task Leader, Westat (301-294-3946)
Denise Glover, Project Director, Westat (301-251-2269)
Diane Ward, Operations Manager, Westat (301-251-4309)
Eric Jodts, Telephone Center Operations Manager, Westat (301-610-8844)
David Morganstein, Statistician, Westat (301-251-8215)
Gary Shapiro, Statistician, Westat (301-517-8028)
Appendix A – Description and Justification of Survey Items
It is important to collect information on all attempts to transfer courses (whether successful or unsuccessful) in order to develop a complete picture of the ease or difficulty of transferring courses. However, there may be some students who decide themselves to opt out of transferring credits (e.g., because of a switch to an entirely new area, so that the student’s past studies do not seem relevant); in that case, their new institutions never face the choice of accepting or rejecting transfer credits. Data on such opting out (and the reasons for it) are important because opting out also results in increased costs and delayed entry into the workforce (see questionnaire item 1).
Past research indicates people are much more accurate when providing general descriptions of their educational experience than when providing specific information such as the number of courses in a particular area. For this reason, the questionnaire asks for the number of courses using a few broad categories rather than asking for specific numbers. It asks students to respond in terms of either the number of courses rather than the number of credits because we expect the number of courses to be easier to remember. Focusing on the number of courses also avoids the complication of whether the number of credits may have changed simply through the conversion of quarters to semesters, rather than through a decision to deny credit (see questionnaire item 2). In case some respondents can remember only the number of credits, the questionnaire includes instructions to help respondents to convert that number into the number of courses.
To measure students’ success in transferring courses, the questionnaire asks for two kinds of responses: a general response that does not depend on course counts (i.e., all of the courses were accepted, some were accepted, or none were accepted), along with counts using broad categories. The first question helps to introduce the topic to respondents, simplifies the questionnaire for those students whose courses were all treated in the same way, and is less subject to memory errors. The use of broad categories of counts allows for the fact that students may have difficulty giving precise answers (see questionnaire items 3 and 4).
ED has a particular interest in knowing whether CTE courses are treated differently than other courses. The questionnaire therefore examines this topic by asking for students’ experience in transferring both CTE and general academic courses (see questionnaire item 5). It is possible that institutions may not accept certain introductory courses for transfer because they are not considered to be at the college level. This category may be worthy of special attention, though it also may be difficult to define. Instead, we chose to address this issue when asking for reasons the transfer courses were not accepted (see questionnaire item 6f).
The questionnaire asks students for the reasons courses were not accepted for transfer, using a series of yes/no items. Potential reasons include a limit on the total number of courses/credits that could be transferred, low course grades, general problems relating to the previous institution, problems relating to the course subject area, and courses that are designed to reflect institutional emphases and that therefore must be taken at the same institution (see questionnaire item 6).
Institutions may charge some type of fee for processing/accepting transfer courses (see questionnaire item 7). Students also face additional costs due to the need to repeat courses, take extra courses, and possibly delay graduation (see questionnaire items 12 to 14.) The questionnaire asks both about the frequency of such costs and the level of the costs. Question 8 asks about the nature of any repeated or extra courses (8a), how much the student learned form repeating any courses (8b), and how the student’s grades compared when courses were repeated (8c). Question 9 asks about extra expenses, and question 10 asks about delays to the student’s graduation.
Personal information is typically collected last, so that respondents will already feel involved in the questionnaire. Many types of personal information will presumably be available through the FAFSA, so there is no need to collect the same information again on this questionnaire (especially since the questionnaire will be limited in length). For example, information on gender and age will be available. However, the FAFSA does not include race/ethnicity, and there may be important differences in course-taking patterns or choices of institutions that lead to differences in transferring credits by race/ethnicity (see questionnaire item 12). FAFSA also does not include information on students’ course of study. Often students at 2-year institutions have not chosen a course of study, and the questionnaire is already designed to collect information on whether the students were successful in transferring CTE and general academic. What may be important, however, is whether students’ transfer reflects a change in their course of study. That is, the greater the change that students made in their education/career plans, the less applicable their earlier preparation might be. Students who make the greatest changes may be the least likely to have their transfer credits accepted (see questionnaire item 11).
Finally, question 13 is designed to support the transcript data collection: by directly obtaining the data from the student, or by obtaining a signed permission form (or at least the student’s verbal assent) to collect the data.
Appendix B – Letter to Pell Grant Recipient
Dear Pell Grant recipient:
The U.S. Department of Education wishes to know more about the experiences of students who transfer from one postsecondary institution to another. We want to know if they have trouble in transferring credits they have earned to their new institutions, and if so, what kind of problems they have encountered. Additionally, we would like to review transcripts from the sending and receiving institutions attended by transfer students to provide additional details about the courses and credits requested for transfer.
As a past Pell Grant recipient who transferred, you have been randomly selected for a short survey on this topic. The survey will take approximately 15 minutes to complete. We also are enclosing a transcript permission form for you to sign and return to our contractor, Westat. Westat will pay the institutions for the transcripts. You can respond to the survey in three ways. (1) In a few days, we plan to call you to get your answers over the telephone. Alternatively, if you wish, you can respond immediately (2) by calling the following toll-free number (1-888-487-0511) or (3) by going to the following web site (PellGrantStudy.com) and using the ID shown below:
Website: PellGrantStudy.com
ID: «PIN»
The survey and transcript study are voluntary, and will have no effect on your receipt of Pell Grants. As a way of thanking you in advance for your time, we are enclosing a check for $10. You may complete the survey using any one of the three methods above. We would greatly appreciate your response by September 15, 2008.
There is nothing in the survey that should present any risk to you, but in any case please be assured that your answers will be used for statistical purposes and will not be disclosed, or used, in identifiable form for any other purpose except as provided by law.
Thank you for your cooperation with this survey. Your responses will help us to better serve you and other students in the future.
Sincerely,
Archie Cubarrubia
Office of the Under Secretary
U.S. Department of Education
Appendix C
EMPLOYEE OR CONTRACTOR'S ASSURANCE OF CONFIDENTIALITY OF SURVEY DATA
Statement of Policy
Westat is firmly committed to the principle that the confidentiality of individual data obtained through Westat surveys must be protected. This principle holds whether or not any specific guarantee of confidentiality was given at time of interview (or self-response), or whether or not there are specific contractual obligations to the client. When guarantees have been given or contractual obligations regarding confidentiality have been entered into, they may impose additional requirements which are to be adhered to strictly.
Procedures for Maintaining Confidentiality
1. All Westat employees and field workers shall sign this assurance of confidentiality. This assurance may be superseded by another assurance for a particular project.
2. Field workers shall keep completely confidential the names of respondents, all information or opinions collected in the course of interviews, and any information about respondents learned incidentally during field work. Field workers shall exercise reasonable caution to prevent access by others to survey data in their possession.
3. Unless specifically instructed otherwise for a particular project, an employee or field worker, upon encountering a respondent or information pertaining to a respondent that s/he knows personally, shall immediately terminate the activity and contact her/his supervisor for instructions.
4. Survey data containing personal identifiers in Westat offices shall be kept in a locked container or a locked room when not being used each working day in routine survey activities. Reasonable caution shall be exercised in limiting access to survey data to only those persons who are working on the specific project and who have been instructed in the applicable confidentiality requirements for that project.
Where survey data have been determined to be particularly sensitive by the Corporate Officer in charge of the project or the President of Westat, such survey data shall be kept in locked containers or in a locked room except when actually being used and attended by a staff member who has signed this pledge.
5. Ordinarily, serial numbers shall be assigned to respondents prior to creating a machine-processible record and identifiers such as name, address, and Social Security number shall not, ordinarily, be a part of the machine record. When identifiers are part of the machine data record, Westat's Manager of Data Processing shall be responsible for determining adequate confidentiality measures in consultation with the project director. When a separate file is set up containing identifiers or linkage information which could be used to identify data records, this separate file shall be kept locked up when not actually being used each day in routine survey activities.
6. When records with identifiers are to be transmitted to another party, such as for keypunching or key taping, the other party shall be informed of these procedures and shall sign an Assurance of Confidentiality form.
7. Each project director shall be responsible for ensuring that all personnel and contractors involved in handling survey data on a project are instructed in these procedures throughout the period of survey performance. When there are specific contractual obligations to the client regarding confidentiality, the project director shall develop additional procedures to comply with these obligations and shall instruct field staff, clerical staff, consultants, and any other persons who work on the project in these additional procedures. At the end of the period of survey performance, the project director shall arrange for proper storage or disposition of survey data including any particular contractual requirements for storage or disposition. When required to turn over survey data to our clients, we must provide proper safeguards to ensure confidentiality up to the time of delivery.
8. Project directors shall ensure that survey practices adhere to the provisions of the U.S. Privacy Act of 1974 with regard to surveys of individuals for the Federal Government. Project directors must ensure that procedures are established in each survey to inform each respondent of the authority for the survey, the purpose and use of the survey, the voluntary nature of the survey (where applicable) and the effects on the respondents, if any, of not responding.
PLEDGE
I hereby certify that I have carefully read and will cooperate fully with the above procedures. I will keep completely confidential all information arising from surveys concerning individual respondents to which I gain access. I will not discuss, disclose, disseminate, or provide access to survey data and identifiers except as authorized by Westat. In addition, I will comply with any additional procedures established by Westat for a particular contract. I will devote my best efforts to ensure that there is compliance with the required procedures by personnel whom I supervise. I understand that violation of this pledge is sufficient grounds for disciplinary action, including dismissal. I also understand that violation of the privacy rights of individuals through such unauthorized discussion, disclosure, dissemination, or access may make me subject to criminal or civil penalties. I give my personal pledge that I shall abide by this assurance of confidentiality.
Signature
| File Type | application/msword |
| File Title | School Survey on Crime and Safety (SSOCS) |
| Author | Brad Chaney |
| Last Modified By | DoED User |
| File Modified | 2008-08-20 |
| File Created | 2008-08-20 |