OMB Memorandum

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: June 6, 2011

FROM: Elise Miller, NCES

TO: Shelly Martinez, OMB

THROUGH: Kashka Kubzdela, NCES

SUBJECT: Focus Groups with High School Seniors and Guidance Counselors, Consumer Information on the College Navigator Website
(OMB# 1800-0011 v.163)

Submittal-Related Information

The following material is being submitted under the U.S. Department of Education’s Generic Clearance of Customer Service Satisfaction Collections (OMB# 1800-0011). This generic clearance provides for the Department to conduct surveys and other studies to evaluate customer satisfaction. The request for approval described in this memorandum is to conduct focus groups with graduating seniors and high school guidance counselors, the results of which will lead to revisions to the College Navigator website that will allow students to compare key data elements on postsecondary institutions. The subject of the focus group is the information students want and need to have when they search for and compare colleges in order to make informed decisions about which institutions to apply to and attend.

Background

College Navigator, developed by NCES in 2007, is a free web-based consumer information tool designed to help students, parents, high school counselors, and consumers search for and obtain information about more than 7,000 Title IV postsecondary institutions in the United States. Among the many data elements about institutions available through this data tool are the following:

• Admissions criteria,

• Programs offered,

• Institution type,

• Retention and graduation rates,

• Prices, including net price,

• Aid available,

• Degrees awarded,

• Campus safety,

• Size of student population and student-to-faculty ratio, and

• Accreditation.

Under a separate task order contract for NCES in 2005, Westat conducted several focus groups with primarily low-income, first-generation students, their parents, and guidance counselors who worked in low- to moderate-income high schools to help refine and redesign the predecessor to the College Navigator website, a website called College Opportunities Online (COOL). Based, in part, on the data collected and analyzed in these groups, NCES redesigned the COOL website. Findings from these groups also provide insights for the development of College Navigator website.

Since August 2008, NCES has added more information to College Navigator as required by the Higher Education Opportunity Act. With the addition of these new items such as net price and 200% graduation rates, there is a great deal of good information for students to use in their college comparison. However, NCES does not want to overwhelm students with information when they first enter the site. Therefore, NCES wants to use the results from these focus groups to help identify a subset of data elements students and guidance counselors deem critical as a “first look” at an institution before further entering into the site for additional information. To accomplish this task, NCES is interested in testing a draft profile containing data elements from seven different categories (i.e., basic information about the college, undergraduate admissions, student financial aid, estimated student expenses before and after aid, student characteristics, student persistence and success, and information about degrees and certificates). The purpose of conducting these focus groups is to determine which elements are of most interest to consumers and why, the extent to which they understand and accurately interpret the terminology and the data presented, and how these data can be used when comparing institutions of interest. After discussing in detail the data elements contained in each topic, which will include items presented in the draft profile and a few additional items on each topic found in College Navigator, each focus group participant will be asked to rank the elements within each category on how important they are to the student or counselor.

The information collected and synthesized in the focus groups will be summarized in a background paper that will be shared with the NCES and the National Postsecondary Education Cooperative, the IPEDS research and development group, and core of the IPEDS Technical Review Panel. NCES will use the background paper to enhance the functionality of the website so that consumers can compare a meaningful subset of key data elements.

NCES recently conducted a single consumer information-related focus group to solicit input from students about proposed information related to the recent gainful employment regulation on disclosures that institutions must present on their websites. This similar project yielded insightful information from students on how they perceived the information and how it could be better presented to them. We hope to gain similar insights through these focus groups. One-on-one interviews were considered as an alternative. However, due to budget and time constraints, a focus group method was chose. However, these focus groups are just the first step in a series of ways in which NCES plans to gain feedback on College Navigator. Other efforts such as usability testing are planned in the next fiscal year. One-on-one usability testing was previously conducted for College Navigator when we initially launched the site in September 2007. As we continue to make changes to the site, we plan toi re-visit that method of gathering feedback as well.

Design and Context

The current request is for approval to conduct three focus groups in June 2011. The summary background paper based on the results will be prepared in July 2011. Westat, the contractor, has drafted focus group materials that include questions about the draft profile containing a subset of items for review by students and guidance counselors. Westat will use a moderator’s guide to ask the same or parallel questions of all participants.

Westat will draw focus group participants from the DC metropolitan area since the main Westat office is located in Rockville, Maryland. The two student protocols are identical, with a few minor modifications: one group will be held with graduating seniors planning to attend 4-year institutions and the second student group with graduating seniors planning to attend
2-year institutions. A separate protocol was developed for use with high school guidance counselors. Approximately 12 participants will be recruited for each focus group in the hope that 9 will show up to participate.

Participants will be recruited by a professional focus group recruiting facility located in Bethesda, Maryland. Recruitment will be based on screening questions prepared by Westat (Attachment I) to determine eligibility for the groups. Attachment II provides additional detail about recruitment procedures, including sample fliers and confirmation e-mails. Attachment III contains the focus group protocols: one for the 4- and 2-year student groups and a second one for the guidance counselors. Attachment IV contains the draft screen shots of the NCES profile as well as screen shots of relevant data elements from the College Navigator (not included in the profile) that will be discussed in the focus groups. Attachment IV also contains two versions of the ranking forms: one for students planning to attend 4-year institutions, which will also be used for the focus group with guidance counselors, and a second one for students planning to attend
2-year institutions. Attachment V contains the Assurance of Confidentiality (consent forms), and Attachment VI contains the Affidavit of Non-disclosure form that Westat staff assigned to the focus groups task have signed.

The focus groups will be held in a facility that is centrally located, easily accessible by car and public transportation, and allows for professional audio recording. Sessions will be held at times convenient for workers’ and students’ schedules. Each focus group, of approximately 90 minutes’ duration, will be conducted by Westat researchers with extensive experience in moderating focus groups of youth and adults. (The moderator for the College Navigator groups also conducted the COOL focus groups in 2007.) A Westat note-taker will attend each of the groups and take notes from behind the one-way mirror. The audio recordings will be made available to Westat for use in fleshing out the notes.

The focus group questions are organized into three sections: (1) a section of warm-up and background questions, (2) the main section containing questions about the specific data elements in each of the seven categories followed by a ranking exercise for each topic, and (3) a concluding section in which all the categories are reviewed and participants are asked to rank the importance of all the categories followed by a few general questions about the format and presentation of the profile.

The first set of background questions is designed to solicit students’ experiences in conducting searches that will provide context for the remaining discussion. Counselors will be asked about their experiences in assisting students with the information gathering and interpreting process.

The second and main body of the protocol will focus on asking questions about each set of data elements by category. Although the discussion will focus on the elements contained in the draft profile for which screen shots will be provided, additional elements within each topic area will also be briefly reviewed (and screen shots will be provided). For example, the draft NCES profile contains a graphical presentation of the numbers of applicants, admitted students, and enrolled students for fall 2009. However, other data provided in making undergraduate admissions decisions (e.g., high school GPA or Test of English as a Foreign Language) and SAT and ACT test score ranges, which are not depicted in the graphical profile, will be discussed using screen shots. In this second section of the protocol, the moderator will ask students about their understanding, use, and the importance of each data element. Typical follow-up questions and probes will ask students to explain a term or concept to ensure they understand what is being asked of them. For example, a question in which students are asked if they understand the differences among grants, scholarships, and loan aid will be followed up with a question asking students to explain the differences as they understand them. After discussing the data elements associated with each topic area, students and counselors will be asked to individually rank the importance of the 3-4 data elements within each category using a ranking form.

Also in the main section of the protocol, guidance counselors will be asked questions that draw upon their experiences working with students. The questions will focus on which elements contained in the profile counselors currently emphasize (and which ones they do not) when helping students conduct college searches, how they use them, and why they deem them important (or not). Guidance counselors will also be asked questions about what information they think students and parents should focus on that does not appear in the draft profile when comparing their choice set of institutions. Like students, counselors will also rank the data elements discussed within each of the categories; five categories total. In the concluding section of the protocol, the moderator will quickly recap the categories discussed and ask consumers to rank the importance of the categories. Finally, participants will share their opinion about the format and presentation of the data as displayed in the profile.

The focus groups will provide an opportunity to enhance the usefulness of the College Navigator tool to make it more meaningful for students, parents, and consumers to compare a critical subset of data elements across several institutions. The focus group protocols are included in Attachment III.

Assurance of Confidentiality

Focus group participants will be informed that their participation is voluntary and that the information they give us will be combined with the responses of others in a summary report that does not identify anyone as an individual, and that their answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573] (see Attachment V). The signed consent forms will be kept separately from the interview files in a locked cabinet for the duration of the study and will be destroyed after the background paper is completed and finalized. The student consent process will include procedures for obtaining parental permission and student permission to audio record the discussion. Similarly, the guidance counselor consent process includes procedures for obtaining permission to participate in the group and audio record the discussion. Contractor staff have in all instances signed the Affidavit of Non-disclosure (Attachment VI).

Project Schedule

Schedule for College Navigator focus groups

College Navigator Group	Start date	End date
Recruitment for three focus groups		6/28/2011
Conduct three focus groups	6/28/2011	6/29/2011
Finalize notes	6/30/2011	7/8/2011
Organize, synthesize, analyze data	7/11/2011	7/15/2011
Draft report	7/182011	7/26/2011
Finalize report based on NCES review	8/2/2011	8/4/2011

Estimate of Hour Burden

Three focus groups are planned with nine participants for each focus group. Each focus group is expected to last approximately 90 minutes. The focus group burden is therefore 40.5 hours, exclusive of travel time. However, there is also a screener, which is estimated to be taken by about 72 individuals and will take on average of 5 minutes to complete; this constitutes 6 hours, for a total burden of 46.5 hours.

Response Type	# Respondents	# Responses	Burden Minutes per Respondent	Total Burden Hours
Screener	72	72	5	6
Focus Group	27	27	90	41
Total	72	99	-	47

Estimate of Costs for Recruiting and Paying Respondents

To thank the respondents for their travel to the focus group facility, time, and effort, the participating students will receive $50, and the guidance counselors, who are typically more difficult to recruit for focus groups, will receive $75.

Cost to Federal Government

The cost of conducting the focus groups will be $82,000, under the Westat subcontract to Coffey Communications, including recruitment and coordination with the focus group facility, facility rental, protocol development, preparation of focus group materials, analysis, report writing, and participant incentives.