Customer Sat Survey for Schools Participating in the Qua

Customer Satisfaction Survey

For Schools Participating in the

Quality Assurance Program

Federal Student Aid, U.S. Department of Education

March 2005

Dear QA Program Participants:

In June 2004, FSA introduced Phase 1 of the web-based version of the ISIR Analysis Tool, and in October 2004, Phase 2 of the Tool was introduced. Using WebEx Technology, QA Program staff conducted training on Phase 1 which included uploading flat ASCII SSN files into the Tool. We later hosted training sessions at both EAC conferences on how to use Phase 2 of the Tool - analyzing your sample data. To continue to improve our service to you, we are asking for your input via a Customer Satisfaction Survey. Your responses to this survey will help us enhance future versions of the ISIR Analysis Tool, the sampling process, and the training and support we provide to you.

The first section of the survey focuses on the training and support that we provided – the Sampling Worksheet, the WebEx sessions on uploading data, and hands-on sessions at the Electronic Access Conferences. The second section of the survey focuses on your experience with completing your sampling and analysis activities.

This survey may require input from various members of your team. As you begin filling out the survey, you will be able to save your entries by pressing “Save and go to the next page.” This will allow you to consult with your team, if need be, and complete the survey in multiple sessions. Press “Submit" on the last page only when you have completed the entire survey. Please note that the survey will only be open for 3 weeks. Since we’d like to get 100 percent participation, please take a few minutes to complete it. Your feedback is important to us.

If you have questions, please contact your QA Representative. Thank you for your support and assistance.

QA Team

Training and Support You Received in 2004

In this section, we are interested in your experiences in the training and the support that QA Program provided in 2004.

Sampling Worksheet

1. Please rate the Sampling Worksheet website which helped to track the milestone activities on the following dimensions:

a) Help in determining when to draw your sample(s)

(poor) 1 2 3 4 5 (excellent)

b) Help in determining when to collect verification data on sampled applicants

(poor) 1 2 3 4 5 (excellent)

c) Help in determining when to upload ISIR information into the tool

(poor) 1 2 3 4 5 (excellent)

d) Help in determining when to conduct your analysis

(poor) 1 2 3 4 5 (excellent)

e) Help in keeping track of progress through out sampling exercise

(poor) 1 2 3 4 5 (excellent) 6 Cannot rate, my school did not return regularly to enter dates as activities were completed

f) General navigation and usability

(poor) 1 2 3 4 5 (excellent)

2. If you have any suggestions about how we can improve the Sampling Worksheet for future use, please provide this information in the space below (e.g., provide training before asking schools to complete, send more reminders, don't send reminders, etc.).

_________________________________________________

WebEx Sessions

3. To assist schools in using Phase 1 of the new ISIR Analysis Tool, FSA conducted a number of training sessions over the summer using WebEx technology. Did your school participate in any of these sessions?

1. Yes

2. No [Skip to question 6]

4. [Only ask if yes on 3] Based on your participation in the WebEx training, please indicate whether or not you agree with the following statements.

1. The WebEx application provided an effective training environment. (Strongly disagree) 1 2 3 4 5 (Strongly agree)

2. I would have preferred just a conference phone call instead of the conference call and internet chat format.

(Strongly disagree) 1 2 3 4 5 (Strongly agree)

c) I received my login information for this session in time to use it.

(Strongly disagree) 1 2 3 4 5 (Strongly agree)

d) There were too many participants in my training session.

(Strongly disagree) 1 2 3 4 5 (Strongly agree)

e) The training provided me with enough information to create the flat ASCII files needed to upload data into the Tool.

(Strongly disagree) 1 2 3 4 5 (Strongly agree)

f) The training provided me with enough information to be able to navigate the web application in order to upload ISIR records.

(Strongly disagree) 1 2 3 4 5 (Strongly agree)

g) FSA should continue using Web-based platforms to provide future trainings. (Strongly disagree) 1 2 3 4 5 (Strongly agree)

5. I would suggest improving the Web-based training in the following way:

______________________________________________________

EAC Conference

6. After attending the 2004 EAC (in either Orlando or Las Vegas) hands-on session on generating reports within the ISIR Analysis Tool, I felt that (please select the statement that best fits):

1. I was confident that I could generate all the reports within the Tool.

2. I believed that I might have some problems generating desired reports within the Tool, but I learned enough to be able to figure it out once I got back to my office.

3. I learned bits and pieces about the Tool, but felt I would need additional help from the QA Program staff in order to generate reports.

4. I was still lost on how to generate reports within the Tool.

5. I cannot comment - did not attend the EAC conference.

6. I cannot comment- did not attend the hands-on session at the QA Pre-Conference.

7. I cannot comment – tried but could not attend the hands-on session because room was full.

7. After attending the 2004 EAC conference (in either Orlando or Las Vegas) training on structuring you own analysis using the ISIR Analysis Tool (that is deciding which reports to run and which records to include), I felt that (please select the statement that best fits):

1. I had a good idea of how to analyze my institutional data and needed no further training.

2. I had a general idea of what to do, but would have liked additional guidance or training on advanced topics.

3. I had a vague idea of how to do the analysis, but needed more assistance in selecting which reports to run and what records to include in my analysis.

4. I was at a complete loss about how to conduct my own analysis. That is, I did not know which report to select and which records to include in my analysis.

5. I cannot comment - did not attend the EAC conference.

6. I cannot comment – did not attend the data analysis session.

7. I cannot comment – tried to attend but could not get into the data analysis session because the room was full.

8. Did you visit the PC Lab to see/discuss the ISIR Analysis Tool during the 2004 EAC conference?

1. Yes.

2. No.

3. No, I tried, but PC Lab staff was busy assisting others and I left before being able to talk with them.

9. [For those who answered (a) in 8 only] Please describe your level of satisfaction with your visit to the PC lab.

1. I was completely satisfied.

2. I was somewhat satisfied.

3. I was somewhat dissatisfied.

4. I was completely dissatisfied.

10. [For those who answer (a) in 8 only] Did you have specific questions/topics that you wanted to discuss with the QA Team about the web-based ISIR Analysis Tool in the PC lab?

1. Yes, I came wanting to discuss specific questions.

2. No, I did not have specific questions. I wanted to learn more about the Tool in general or pick up additional materials.

11. [For those who answered (a) in 10 only] Please briefly describe the specific questions/topics that you brought to the PC lab to discuss, e.g., navigational help on generating a particular report, selecting a specific sub-group of students, or guidance on structuring your own analysis from the reports available.

__________________________________________________

12. [For those who answered (a) in 10 only] Please describe the level of help you received from the PC lab staff. (Mark all items that apply).

1. The PC lab staff was able to completely resolve my questions.

2. The PC lab staff was able to resolve some of my questions.

3. The PC lab staff was not able to resolve any of my questions.

4. The PC lab staff was able to get me to the next step, i.e., identify an effective course of action.

5. The PC lab staff was too busy attending to other clients to resolve my questions.

6. The PC lab staff did not have sufficient knowledge to answer some of my questions.

7. The PC lab staff did not have sufficient time to fully address some of my questions, e.g., lab was closing, appointment over.

8. Other comments ____________________

Using the ISIR Analysis Tool

In this section, we are interested in your experiences in drawing the sample, using the web-based ISIR Analysis Tool, and developing plans to address the findings raised from analyzing the data. For the small number of schools that drew multiple samples, please base all of your responses on your first sample.

Drawing the random sample

13. Did you use the ISIR Analysis Tool to draw your random sample?

a) Yes.

b) No, we drew a random sample on our own.

14. Overall, how would you characterize your experiences drawing the random sample of applicants?

a) Smooth, without any problems worth reporting.

b) Some problems, but largely ok.

c) Many problems.

15. [For those answered (b) or (c) in 14 above only] While drawing the random sample, which of the following potential problems did you experience? (Please mark all that apply).

1. The exercise required computer skills that our team cannot provide.

2. The time and effort needed to draw the sample was much more than anticipated.

3. There was insufficient guidance on the best time to draw the sample.

4. There was insufficient guidance on how to draw the sample.

5. There was insufficient guidance on who should be included in the sample.

6. Other (please specify).

____________________________________________

16. Please indicate the month that you completed drawing the sample that was used in your analysis:

Month___

17. With hindsight, I wish I had completed drawing our sample:

a) Earlier than we did.

b) Later than we did.

c) About the time we did.

18. If you would have chosen a different time to draw your sample, please use the space below to explain why.

______________________________________

Verification of sampled applicants

19. Was your data verification process for sampled applicants, part of your institutional verification or a separate effort?

1. Completely integrated into our normal institutional verification process

2. Partly integrated

3. A separate process

20. Overall, how would you characterize your experiences verifying information for sampled applicants?

a) Smooth, without any problems worth reporting.

b) Some problems, but largely ok.

c) Many problems

21. [for those who answered (b) or (c) in 20 above only] When verifying information from your sample, which of the following potential problems did you experience? (Please mark all that apply)

1. The work to track and follow up with the students for verification required more staff time, skill, and resources than anticipated.

2. The number of weeks we needed to track and follow up with the students for verification was much more than anticipated.

3. I wish I had started this process earlier than I did.

4. I wish I had started this process later than I did.

5. Objections from the selected students/parents were greater than anticipated.

6. Some students never complied with turning in their verification documents and therefore, aid offers needed to be withdrawn.

7. Other problems. Please specify

22. Please indicate the month that you completed or cut off the verification process for the sample you analyzed:

Month___

23. With hindsight, how would you do your verification differently?

___________________________________________________

Uploading Data into the web-based ISIR Analysis Tool

24. Which statements describe the level of technical capacity to write ASCII files from your aid processing system? (Check all that apply)

a) We have experience writing data from aid processing systems in various ASCII formats for non-QA purposes.

b) Our school's system for processing aid makes it easy for us to retrieve information in the format required by the Tool.

c) Our school's system for processing aid presents challenges for us to retrieve information in the format required by the Tool.

d) At least one member of our school's QA team has the technical skills required to facilitate transferring information from our system into the Tool.

e) Our team must rely on the technical skills of a financial aid office staff member who generally does not work on QA to facilitate transferring information from our system into the Tool.

f) Our team must rely on someone outside the financial aid office to facilitate transferring information from our system into the Tool.

h) Our team experienced difficulties securing the time of the staff member(s) with the technical capacity to facilitate transferring information from our system into the Tool.

25. Overall, how would you characterize your experiences creating flat ASCII files and uploading ISIRs and user-specified data into the web-based ISIR Analysis Tool?

a) Smooth, without any problems worth reporting.

b) Some problems, but largely ok.

c) Many problems.

26. [For those who answered (b) or (c) in 25] What problems did you experience while uploading ISIRs and user-specified data into the web-based ISIR Analysis Tool? (Please mark all that apply)

1. Instructions on preparing the ASCII file were not clear

2. Took more time and effort to write flat ASCII files in the format required by the Tool than anticipated

3. Had difficulty navigating within the Tool to upload information

4. Felt that more training was needed

5. With hindsight, I wish I had started this step earlier

6. With hindsight, I wish I had started this step later

7. Other (please indicate)

________________________________________

Data analysis

27. Overall, how would you characterize your experiences with generating reports from the web version of the ISIR Analysis Tool?

a) Smooth, without any problems worth reporting.

b) Some problems, but largely ok.

c) Many problems

28. [For those who answered (b) or (c) in 27 only] What problems or reactions did you have while generating reports from the ISIR Analysis Tool? (Please mark all that apply)

1. Experienced difficulty generating the standard reports

2. Experienced difficulty using the drill down feature of the Tool

3. Experienced difficulty using the filter feature of the Tool

4. Experienced difficulty generating Ad Hoc reports

5. Other (please indicate)

29. Overall, how would you characterize your experiences with structuring your school’s analysis using the ISIR Analysis Tool?

a) Smooth, without any problems worth reporting.

b) Some problems, but largely ok.

c) Many problems

30. [For those who answered (b) or (c) in 29 only] What problems or reactions did you have while structuring your own analysis? (Please mark all that apply)

1. Experienced difficulty defining a goal for the analysis.

2. Experienced difficulty deciding which reports to run.

3. Experienced difficulty deciding what records to include in my analysis.

4. Experienced difficulty getting the standard reports available in the Tool to focus on a specific sub-population of interest.

5. Experienced difficulty applying standard reports to my analysis.

6. Experienced difficulty applying Ad Hoc reports to my analysis.

7. Experienced difficulty knowing when the questions we were trying to address could be adequately addressed with the reports we had run, i.e. knowing when to stop running reports.

8. Once I understood the analytic capabilities of the Tool, I wished I had drawn my sample differently (e.g., a larger sample, different time).

9. Other (please indicate).

------------------------------------------------------

31. Please use the grid below to indicate what goals your school had for this year's analysis and whether or not you were able to reach these goals using the Tool. (Mark all that apply)

	Was this a goal for your school?		For all yes responses, please indicate outcome of analysis
Goal	Yes	No	Analysis helped achieve goal	Analysis did not help achieve goal
Improve FAFSA education outreach efforts
Expand school/discretionary verification efforts
Reduce school/discretionary verification efforts
Evaluate a specific aspect(s) of your school/discretionary verification efforts
Other (please, specify)

32A. [For those who answered YES to any item in 31 above and analysis helped achieve that goal] Briefly describe the improvement sparked by analysis.

__________________________________________________

32B. [For those who answered YES to any item in 31 above and analysis helped achieve that goal] When will you begin to implement this improvement?

a) The 2005-06 processing year.

b) The 2006-07 processing year.

c) Other, please specify.

32C [For those who answered YES to any item in 31 above and analysis DID NOT help achieve that goal] Briefly explain why you are not able to use the results generated from the ISIR Analysis Tool.

a) Our analysis confirmed the usefulness of criteria we are already using to select applicants for verification.

b) Using the Tool, we could not get down to the level of detail we needed to identify improvements.

c) There were too few applicable cases in our sample to justify a change.

d) Other, please explain

_____________________________________________________

33. After using the new web-based ISIR Analysis Tool this year, what changes or improvements would you like to see in the application in the future? (e.g., additional standard reports, changes in navigation, additional output formats, etc.)

_________________________________________________

34. After working through the first sampling and analysis exercise with the new Tool, what additional training would you request be offered in the future?

_________________________________________________

13