Supplementary Questions

Supplemental Questions for NASA Usability Data Collections Generic Clearance (OMB Control Number 2700-0127)

1. Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

We are not conducting surveys, but this provides information relevant to our usability studies.

To help us understand product usability and the needs of our education audiences (K-12 and post secondary), we would like to conduct usability testing and focus group sessions in fiscal year 2007 (FY07). We've budgeted for four trips across a widely representative geographic region to provide feedback from target audiences. Conducting the three different studies while at these various locations will help us gather a multitude of data for various projects while saving money by avoiding travel if each study were conducted separately. These projects support NASA Exploration Systems Mission Directorate (ESMD), NASA.gov For Students section, and educational pod-casting products.

We have found a tool on the National Center for Education Statistics Web site that enables us to search for K-12 schools within a certain number of miles from a particular location (by zip code). This will enable us to be more cost effective in terms of time and money by choosing K-12 schools that are within 1-5 miles of the college area. http://nces.ed.gov/globallocator/

Site visits will be coordinated after approval is received from the Office of Management & Budget (OMB), as we will need to work with school administrators and faculty to determine which schools we will visit and to schedule the studies. Due to the end of the school year nearing, some of the studies may have to occur in July and August 2007, the beginning of the 2007-2008 school year. NASA will also collaborate with school districts who offer year-round school to schedule usability studies during the summer to ensure its annual performance goals are met.

We have reviewed NASA strategic framework information and pinpointed target groups for NASA’s education and outreach efforts, and we chose specific regions to provide demographically diverse representation from the education community; however, specific locations may change depending on academic calendars and school administration approval.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? statistics? What suggestions did you get about improving the survey? 

We are not conducting surveys, but we are conducting usability studies involving use of multiple collection instruments during sessions. NASA Education staff and contractors helped develop the collection instruments, including focus group questions, questionnaires, and consent forms. Onsite educators were consulted when developing questions for students. Our staff podcasting expert helped develop the podcasting questionnaires and focus group questions.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.) 

Sessions

Students

Due to the difficulty of pulling a selected few K-12 students from classes to schedule a focus group session, NASA will work with various schools to schedule times to visit classrooms to conduct field studies. The session lengths will vary per district, with the majority being 60-70 minutes long. Sessions will be held to discuss user needs and site content related to the For Students area of NASA.gov and user practices and preferences related to NASA educational podcasts. The podcasting sessions will target 11th -12th graders and undergraduate students.

This approach will enable us to interact with many more students without making any feel left out because they weren’t invited to smaller focus group sessions. Class discussion where students will be asked to provide feedback to targeted questions will require 30-45 minutes. Also, we plan for the students in grades 3-12 and post secondary students to complete a brief usability questionnaire (burden time 15 minutes maximum), but completion of this will be done within the 60-70 minute session.

Sessions at universities will be 60 minutes long and will have 6-7 participants per session. These students will also complete a usability questionnaire (15 minutes) after discussions in the focus group.

Educators

We want to target 60-minute sessions, so we will aim for 6 participants for each session to help ensure each participant has opportunity to respond to each question. We will also try to limit the session to about 5-6 questions to ensure each respondent has roughly 10 minutes of total feedback time. We will conduct sessions for K-4, 5-8, and 9-12 educators.

We will record these sessions, but we're still going over options as to what method we will use (video versus audio). Due to traveling on planes, we may be limited in how much and what equipment we can take. Additionally, specific schools will be determined upon receiving approval from OMB as we will have to consider and work around each individual school district’s policies and time schedules for availability.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

Both focus group sessions and open-ended questionnaires will be used to gather user data from students and educators nationwide. The usability team would be able to interact directly with users during these sessions, and this interaction would help determine how NASA should adapt a site or product to improve its pertinence and usability for customers. Since the data that will be gathered in these usability sessions is opinion-based, little to no statistical compilation or calculation is necessary other than general percentages of students/educators who like/dislike a product or service, or who express a particular opinion. (Ex: Twenty percent of K-12 educators who participated in podcasting focus groups expressed that NASA Student Opportunities podcasts are helpful in providing students information on NASA careers.)

Questions from Section B of the 83-I Supporting Statement

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

We will travel to four U.S. regions. Our respondent count may vary per regional location. Schools are structured differently, and student count per classroom varies. However, we estimate 524 respondents; this figure was determined as follows:

ESMD Focus Groups: Target six participants per focus group session (3)=18 educators per location. 18*4=72 participants*1 hour=72 burden hours

Podcasting Focus Groups: Combined participants for sessions= 32 per location. 32*4=128 participants*1 hour=128 hours

Student Focus Groups: 81 participants per location= 81*4=324*1 hour=324 burden hours

Total participant count: 524

Total burden hours: 524

We are not doing a statistical survey online or offline; however questionnaires will be used during the Student and Podcasting Focus Groups to gather additional opinions and input. But, the questionnaires are not formal surveys. No complex statistical calculations are required, merely figuring percentages of users who respond to questions favorably/unfavorably (ex: 47% of users found the podcasts easy to access) or who express particular opinions. We will not use weighted responses; each user’s response will be considered.

2. Describe the procedures for the collection of information including:

* Statistical methodology for stratification and sample

selection,

* Estimation procedure,

* Degree of accuracy needed for the purpose described in the justification,

* Unusual problems requiring specialized sampling procedures, and

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Focus group questions, questionnaires, observation, and consent forms will be used in these studies. No complex statistical calculations are required, merely figuring percentages of users who express opinions or who respond to questions favorably/unfavorably (ex: 47% of users would like more moon-related science lessons.). Since the data is opinion-based, focusing on user preferences and needs, there are no right or wrong responses. These responses will help us align our products to our target users’ needs, and help improve online services and products. To help reduce time burdens and costs, we will conduct multiple studies in each region for target education audiences: K-12 students, K-12 educators, and undergraduate students. By conducting multiple studies for ESMD, Students, and Podcasting at the same time, we are limiting the number of visits to schools and helping to reduce travel/labor costs.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Participation in NASA Education usability studies is strictly voluntary. Schools within four geographic regions will be contacted, and we will work with administrators to invite students and teachers to participate. Because NASA produces free educational products and online services, the public is generally eager to provide feedback and suggestions.

Reliability of the data is not an issue, as the data gathered is based on users’ opinions and preferences.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the

main collection of information.

n/a

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Due to the simplicity of the calculations used (calculating percentages), no external consultation is needed. Contractors supporting this usability studies for NASA Education include:

Dawn Gaddis (UNITeS/SAIC), Marshall Space Flight Center, Huntsville, AL

Deana Nunley (UNITeS/Analytical), Marshall Space Flight Center, Huntsville, AL

Heather Deiss (UNITeS/SAIC), Marshall Space Flight Center, Huntsville, AL

Kathy Forsythe (UNITeS/SAIC), Marshall Space Flight Center, Huntsville, AL