DLN Focus Group Questions

PAPERWORK REDUCTION ACT

NASA GENERIC CLEARANCE SUBMISSION

OMB CONTROL NUMBER 2700-0129

This form should be used if you are submitting a collection of information for approval under the NASA generic clearance assigned OMB control number 2700-0129. Mail this form, supplemental documentation, the collection instrument, and any additional documentation to: Walter Kit, NASA Paperwork Reduction Act Desk Officer, NASA Headquarters, Washington D.C. 20546, or via e-mail to [email protected].

If the collection does not satisfy the requirements of the program clearance, you should follow the regular PRA clearance procedures described in 5 CFR 1320.

NASA Subagency:

NASA Education

Title (Please be specific):

NASA Education Usability Data Collections—Digital Learning Network (DLN) Focus Groups

Burden Hour/Cost Estimates

Number of respondents: 27 Annual Responses Approved by OMB: 600

Total Burden Hours: 27 Annual Burden Hours Approved by OMB: 900

Cost to participant: 0

Cost to NASA: $6,000

Agency Contact (person who can best answer questions about the content of the submission)

Name: Dawn Gaddis/Caryn Long

Phone: 256-544-0167/757-864-5032

Certification:

The collection of information requested by this submission meets the requirement of the OMB approval for OMB control number2700-0129.

Signature of OMB Program Official

___________________________________________________________________________

Date

_____________________________

Signature of NASA Paperwork Clearance Officer

____________________________________________________________________________

Date

_____________________________

Instructions for Submission under NASA Generic Clearance for Usability Data Collections

1. Consult with the NASA Paperwork Reduction Act Desk Officer to confirm that your collection meets the conditions for submission under this clearance.

2. Link documentation for individual collections to information in the original Supporting Statement.

3. Complete and sign the cover sheet. Note: Any questions during the review about the survey will addressed directly to the person listed as the Agency Contact.

4. Give the cover sheet, supplemental documentation including the data collection instrument, and any other helpful information to the NASA Paperwork Reduction Act Desk Officer, who will review the materials, sign the cover sheet and send the package to OMB. The Desk Officer will be notified when the collection is approved under OMB Control Number2700-0129.

Supplemental Questions for NASA Usability Data Collections Generic Clearance (OMB Control Number 2700-0127)

1. Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

A focus group interviewer stationed at NASA’s Langley Research Center (LaRC) will be conducting these focus groups with teachers who are registered DLN users of schools that we service at three DLN centers: Glenn Research Center (GRC), LaRC, and Johnson Space Center (JSC). The DLN provides free, interactive programs to enable teachers and students to learn more about our home planet and to journey into space using video conferencing and Web casts. Students of all ages are able to speak directly with NASA personnel including scientists, engineers, astronauts, and education specialists. Placing an emphasis on science, technology, engineering, and mathematics (STEM), the DLN offers a wide variety of distance learning events designed to educate through demonstrations and real-time interactions with experts. All programs are aligned to national standards with pre- and post-activities to enhance video conferencing and Web casting experiences.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? statistics? What suggestions did you get about improving the survey? 

The survey was conducted with cooperation between focus group experts at Wheeling Jesuit University’s (WJU’s) Classroom of the Future (COTF) and the DLN coordinators at GRC, LaRC, and JSC at the winter professional development conference in New Orleans. Members of this committee studied requested program trends and brainstormed questions that would give the DLN information that would benefit content development needs for schools across the nation. DLN coordinators relied on the focus group experts to fine-tune the language of the questions in order to validate the responses across the centers.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.) 

The sessions will be conducted via DLN technology. A representative group will be combined for each center to participate in the focus group. Three groups consisting of users that are new to the DLN, users that are in their second year, and veteran users will be interviewed within their classifications. Elementary, middle, and high school educators with a variety of school backgrounds will be included. The NASA Education Evaluation Information System (NEEIS) will provide the information we need in order to insure a diverse school background selection from chosen subjects derived from the DLN’s registration area of the administrative site. A goal of 80% of the customers asked will participate in the survey. DLN coordinators will contact their customers personally to ask for their participation.

Sessions

Educators

We want to target 45- to 60-minute sessions, so we will aim for three participants for each session to help ensure each participant has opportunity to respond to each question. We will also limit the session to five questions to ensure roughly 10 minutes per question of total feedback time. Sessions will be conducted for a mix of K-4, 5-8, and 9-12 educators. In order to be time efficient, questions will be sent to participants in advance to allow for preparation of answers.

We will record these sessions via video as they will be conducted through the DLN. Additionally, specific schools will be determined upon receiving approval from OMB as we will have to consider and work around each individual school district’s policies and time schedules for availability. The sessions will be conducted during the third quarter of fiscal year 2008.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

The focus group sessions will be used to gather user data from educators nationwide. The usability team will be able to interact directly with users during these sessions, and this interaction will help determine how NASA should share the DLN modules to improve their pertinence and usability for customers. Since the data that will be gathered in these usability sessions is opinion-based, little to no statistical compilation or calculation is necessary other than general percentages of educators who like/dislike a product or service, or who express a particular opinion. (Ex: Twenty percent of K-12 educators who participated in DLN focus groups expressed that NASA DLN modules are helpful in providing students information on NASA careers.)

Questions from Section B of the 83-I Supporting Statement

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The potential respondent universe will be stratified groups of teachers who have one or more years experience with teaching through video conferencing. The teachers will be interviewed in homogenous groups for the pilot study. A voluntary group of teachers will be divided into groups of first-year users, second-year users, and veteran (third-year) users.

Sample	Number of Participants
First-Year Users	9
Second-Year Users	9
Veteran Users	9

NOTE: Number of participants is just an estimate

2. Describe the procedures for the collection of information including:

* Statistical methodology for stratification and sample

selection,

* Estimation procedure,

* Degree of accuracy needed for the purpose described in the justification,

* Unusual problems requiring specialized sampling procedures, and

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Focus group questions and consent forms will be used in these studies. No complex statistical calculations are required, merely figuring percentages of users who express opinions or who respond to questions favorably/unfavorably (ex: 47% of users would like more moon-related science lessons.). Since the data is opinion-based, focusing on user preferences and needs, there are no right or wrong responses. These responses will help us align our products to our target users’ needs, and help improve distance learning services and products. To help reduce time burdens and costs, we will conduct multiple studies in each region for target education audiences: K-12 educators. By conducting these visits via the DLN, we are eliminating travel costs.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Participation in NASA DLN usability studies is strictly voluntary. Schools within three geographic regions will be contacted, and we will work with administrators to invite teachers to participate. Because NASA produces free educational products and online services, the public is generally eager to provide feedback and suggestions.

Reliability of the data is not an issue, as the data gathered is based on users’ opinions and preferences.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

n/a

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Due to the simplicity of the test focus group, no outside assistance is needed. Civil servants and contractors supporting this pilot focus study for the NASA Digital Learning Network include:

Dr. Robert M. Starr (Project Manager/DLN), Langley Research Center, (Hampton, VA)/757-864-9492

Caryn Long (Assistant Manager/NASA DLN/Oklahoma State University), Langley Research Center, (Hampton, VA)/757-864-5032

Gamaliel Cherry (Instructional Presenter, NASA DLN), Langley Research Center, (Hampton, VA)/757-864-1279

Elmar Koldert (co-op student), Langley Research Center, (Hampton, VA)/757-864-3462

Focus Group Procedure

Anonymity of data responses after collection will be ensured from the focus group facilitator prior to starting the focus group. Specific names and other confidential information will not be shared or published without consent. Given the medium, the facilitator will disseminate copies of the focus group questions to the participants before starting the actual focus group session. Each session will be recorded and transcribed for data analysis.

Script

• Good morning/afternoon!

• Thank you for taking the time to meet with us. We will honor your time by making sure that we wrap up in the next 60 minutes.

• Does anyone mind if we tape record this for our records? We won’t share the tapes.

• Our evaluation is qualitative. This means that our primary point is to gather information that helps the DLN fine-tune and improve its ongoing efforts; and that the information we are collecting is by design descriptive rather than numeric. We want your words and your impressions about our programs.

• All information we collect is confidential as to who provided it. For example, we will not disclose who actually participated in this focus group nor will our final report make any attributions for quotes. We hope this encourages you (if you need encouragement) to speak freely.

• Our evaluation will result in a written report later this year. This report will be delivered to the appropriate NASA officials, and it is our understanding that they intend to share this data with a larger constituency.

Directions:

“Please answer each of the following questions to the best of your knowledge pertaining to your experience with NASA’s Digital Learning Network.”

Focus Group questions:

1) How did the pre/post activities hinder/help the module presentation? How consistently do you use them?  If not, what keeps you from using these tools?

 

2) How do the DLN events help you meet STEM and instructional technology integration requirements or mandates?  Explain.

 

3)  Describe the level of interactivity your students have with our presenters during DLN events.

 

4)  What value do these DLN events and this type of delivery have for your students?

 

5)  What level of interest have your students shown in STEM careers after participation in a DLN event?  Give some examples.