OMB Clearance_Leadership and Management Seminar Evaluation

DOCUMENTATION FOR THE GENERIC CLEARANCE

OF CUSTOMER SERVICE SATISFACTION COLLECTIONS

TITLE OF INFORMATION COLLECTION: Leadership and Management Seminar Follow-Up Evaluation

[X] SURVEY [ ] FOCUS GROUP [ ] SOFTWARE USABILITY TESTING

DESCRIPTION OF THIS SPECIFIC COLLECTION

BACKGROUND

Under Title IV of the Higher Education Act of 1965, which covers the administration of U.S. federal student financial aid programs, Federal Student Aid (FSA) administers the following programs: Pell grants, Stafford loans, PLUS loans, and the “campus-based” programs including Federal Work Study, Perkins Loans, and Federal Supplemental Educational Opportunity Grants. Federal regulation requires schools to have written policies and procedures for the administration of the Title IV student assistance programs. To ensure institutional regulatory compliance, FSA provides training and technical assistance for financial aid administrators, institutional leaders, and other institutional support staff nationwide via its Information for Financial Aid Professionals (IFAP) website and the School Experience Group (SEG). The SEG’s mission is to identify and provide the unique service needs of all post-secondary education institutions. Within SEG, the Minority Serving and Under Resourced Schools Division (MSURSD) is responsible for providing support, assessment, and training specifically targeted for Minority Serving Institutions (MSIs).

FSA is working to expand the SEG’s program and support offerings to enhance the services available for MSIs. A new service offering under the purview of MSURSD is the 2012 Leadership and Management Seminar which aims to:

• increase awareness among institutional presidents/CEOs, financial aid directors, and other executives of the role they play in the Title IV programs at their institution;

• enhance institutional governance, management and compliance of Title IV Programs and thus help institutions avoid financial risks;

• provide a forum for sharing with and learning from peers; and

• discuss effective leadership and management competencies.

INTENDED PURPOSE AND NEED FOR THE COLLECTION

FSA has contracted Windwalker Corporation to evaluate and report on the long term effectiveness of its 2012 Leadership and Management Seminar, which FSA plans to conduct annually. Windwalker is creating a long-term (2 year) evaluation framework and monitoring plan that can be administered to seminar participants and provides a formal system to measure the short- and long-term effectiveness of the seminar as well as identify areas for improvement. The evaluation includes the development of a web-based survey for presidents/CEOs, financial aid directors, and other executives who participated in the seminar and will provide a first look at the seminar’s impact, such as where and how schools are retaining and/or using information learned during the seminar and whether any policy, operational, and/or organizational changes have been made as a result of participation in the seminar.

The web-based survey consists of 17 items to collect feedback from seminar participants. Specifically, the survey will collect data from presidents/CEOs, financial aid directors, and other executives on:

• perceived value and utility of the seminar;

• retention and application of knowledge regarding Title IV requirements;

• growth in understanding their role in ensuring compliance;

• any policy, operational, and/or organizational changes they have implemented as a result of attending the seminar; and

• ideas regarding how to improve the seminar or make it more relevant to their work.

The survey is a component of a larger evaluation effort to guide SEG’s program development and enhancement of services available for MSIs. The survey instrument and email communications (described below) are provided as separate attachments.

COLLECTION PROCEDURES

The web-based survey will be developed using SurveyMonkey, a survey platform and data analysis tool. Using this platform allows for the development of a computer-based survey of varied question types (e.g., questions using Likert scales and open-ended questions) to be administered to respondents via the delivery of a web link through an email message. Both closed- and open-ended items are included in the survey to obtain quantitative and qualitative data to aid in the effectiveness assessment of the seminar. Specifically, nine items are close-ended, seven items are open-ended, and an additional item has both close-ended and open-ended components. Respondents will receive at most seven open-ended items (due to skip logic) and three of the open-ended items are optional. The survey is expected to take respondents approximately 10 minutes to complete. Respondents will use radio buttons or check boxes for close-ended items to indicate their response(s), and text fields will be provided for the open-ended items. Skip logic is utilized for three questions, such that a respondent’s answer to a close-ended item will affect which follow-up open-ended item is displayed. This mixed-methods approach uses the closed-ended responses to provide quantitative information on themes already identified as seminar participation outcomes, while open-ended responses allow Windwalker to gather qualitative data, allowing for more detailed findings and the discovery of themes not represented in the closed-response items.

Once the survey is finalized and ready to deploy, FSA will provide Windwalker a list of names, email addresses, and demographic variables of interest (e.g., institution name and participant’s position title) of seminar participants to whom the survey link will be sent. Each participant will receive an email from an FSA point of contact via surveymonkey.com with an invitation to complete the survey. The email invitation will include details on the purpose of the survey, the anticipated length of time to complete the survey, and the survey deadline date. The survey will remain open for two weeks, and two reminder emails will be sent subsequent to the launch email to participants who have not yet completed the survey. The first reminder will be sent five business days after the survey launch, and a second reminder will be sent eight business days after the survey launch.

Once the survey has closed, Windwalker will begin data analysis. Data will be captured by the SurveyMonkey tool, allowing for download into IBM SPSS Statistics, a software package used for statistical analysis. SurveyMonkey’s text analysis software will be used to analyze the open-ended responses. Where applicable, survey data will be compared to the pre- and post-seminar evaluation survey data previously collected by FSA. Open-ended responses will be coded by theme and the frequency counts for comments by theme will be provided in a December report (see the “Planned Use of Data” section). Should the number of comments for any given open-ended response exceed SurveyMonkey’s text analysis software’s minimum threshold of 20 comments, a word cloud will be created for each open-ended survey question, providing a visual representation of the frequency of the most popular terms. All coding rubrics will be provided to FSA.

DATES, LOCATIONS, AND PARTICIPANTS

The proposed timeframe for data collection is November – December, 2012 and will be conducted entirely online. No focus groups are planned and no payments, stipends, or incentives are proposed. Windwalker will administer the survey to the population of 111 presidents/CEOs, financial aid directors, and other executives who participated in the 2012 Leadership and Management Seminar. This timeline meets the contractual requirement that the survey be administered within three months after FSA provides baseline data from the 2012 Leadership and Management Seminar (projected date of receipt approximately November 15, 2012) and after OMB clearance is obtained. The survey will also be shared with MSURSD/FSA leadership for approval.

PLANNED USE OF DATA

The survey results will provide critical, primary information to evaluate the effectiveness of the 2012 Leadership and Management Seminar, such as whether presidents/CEOs, financial aid directors, and other executives feel they have retained and applied knowledge gained through the seminar, and whether they have instituted any policy, operational, and/or organizational changes as a result of attending the seminar. Where possible, findings from the survey will be compared to the data gathered from the pre- and post-seminar evaluations to determine if there are gains or losses in perceived seminar utility once presidents/CEOs, financial aid directors, and other executives return to their institutions.

A report to be delivered at the end of December to FSA will include a summary of findings from the Leadership and Management Seminar Survey. A goal of the report is to identify what, if any, changes presidents/CEOs, financial aid directors, and other executives are planning as a result of their seminar attendance. The report will also discuss the perception of presidents/CEOs, financial aid directors, and other executives regarding the utility of the seminar and whether they have been able to remember and apply information learned during the seminar.

Open-ended responses will be coded by theme and the frequency counts for comments by theme will be provided in the December report. As previously mentioned, should the number of comments for any given open-ended response exceed SurveyMonkey’s text analysis software’s minimum threshold of 20 comments, a word cloud will be created for each open-ended survey question, providing a visual representation of the frequency of the most popular terms.

AMOUNT OF ANY PROPOSED STIPEND OR INCENTIVE

Not applicable.

BURDEN HOUR COMPUTATION (Number of responses (X) estimated response or participation time in minutes (/60) = annual burden hours):

Category of Respondent	No. of Respondents	Participation Time	Burden
Presidents/CEOs, financial aid directors, and other executives	111	10 minutes	18.5 hours
Totals	111	10 minutes	18.5 hours

BURDEN COST COMPUTATION

Category of Respondent	No. of Respondents	Hourly Rate1	Response Time	Total
Presidents/CEOs	76	$84.88	10 minutes	$1,075.40
Financial aid directors	1	$57.91	10 minutes	$9.65
Other executives	34	$55.04	10 minutes	$311.78
Totals	111		10 minutes	$1,396.83

STATISTICAL INFORMATION

A census of all 2012 Leadership and Management Seminar participants will be used to select participants for the survey. Customer satisfaction surveys typically have a response rate of 10-15%. However, because the targeted survey respondents are presidents/CEOs, financial aid directors, or other executives of postsecondary institutions, they may exhibit an above-average level of engagement in the subject matter of the survey. Therefore, response rates may be as high as 30-50%. Descriptive statistics will be used to analyze the quantitative data, and SurveyMonkey’s text analysis software will be used to analyze the open-ended responses.

REQUESTED APPROVAL DATE: 10 business days past submission date

NAME OF CONTACT PERSON: Chris Lemmie

TELEPHONE NUMBER: (202) 377-3225

MAILING LOCATION: 830 First St. N.E., Washington, DC 20202

ED DEPARTMENT, OFFICE, DIVISION, BRANCH: Chief Customer Experience Office,

Office of Federal Student Aid, U.S. Department of Education

1 Source: May 2011 National Occupational Employment and Wage Estimates, Occupational Employment Statistics, United States Bureau of Labor Statistics.

5