0693-0043 TRANSTAC - SupportingStatement

Department of Commerce (DOC)/National Institute of Standards and Technology (NIST)

Usability Data Collection Generic Collection

OMB Control # 0693-0043

Expiration Date: 10/31/2012

Title of Collection: Spoken Language Communication and translation System for Tactical Use (TRANSTAC) Foreign Language Expert Survey

Supporting Statement

1. Explain who will be surveyed and why the group is appropriate to survey.

As part of a technology evaluation of speech-to-speech (foreign) language translation systems, evaluation study participants will be surveyed. The participants will be selected based on expertise in the languages chosen to test the translation systems.

There will be two evaluations, approximately 3 months apart. Four translation systems will be evaluated during each of the evaluation periods. Each evaluation will have 10 subjects. The first evaluation will have Pashto as the selected foreign language. Iraqi Arabic is the selected language for the second evaluation. The selected foreign language is denoted in each survey instrument as “<focus-language>”.

Attachment A provides the spreadsheet showing the calculated burden hours.

2. Explain how the survey was developed including consultation with interested parties, pretesting, and responses to suggestions for improvement.

The questionnaires that have been developed are based on questions used during previous phases of the program in which six evaluations were conducted, and standard templates used by our usability and language translation groups. The demographic and usability questionnaires are composites of questions from these sources, as well as, incorporating feedback and suggestions from the previous studies and the system researchers whose translation systems will be evaluated. We believe the forms are highly usable in their current state.

3. Explain how the survey will be conducted, how customers will be sampled if fewer than all customers will be surveyed, expected response rate, and actions your agency plans to take to improve the response rate.

Prior to the evaluation, all Foreign Language Expert (FLE) subjects will be given a form that explains the study and the expected participation for each Foreign Language Expert in the study, see Attachment B. He or she will be given a paper copy and asked to complete the Demographic Information survey questions. As part of the study, survey completion is required; therefore, the expected response rate is 100%. All subjects will receive training on how to appropriately perform a dialog-interaction for the type of automated translation devices being tested.

Each subject will receive training on how to interact with the translation system prior to the test period in which that particular system will be tested. Each subject, under direction of the test conductor, will perform his or her assigned scenarios for each translation system tested. Immediately following each scenario, the Post-Scenario Survey will be administered to each participant. The expected response rate is 100% since each participant will be provided the survey by the test conductor and will complete the survey as part of the overall evaluation procedures.

4. Describe how the results of the survey will be analyzed and used to generalize the results to the entire customer population.

The demographic information will be used to characterize the subject population used in each study overall and potentially allow the ability to bin subjects into categories based on demographic characteristics. Further, the demographic data could be used to suggest causality of possible outliers in the dialog data provided to the translation systems, e.g., the language influence of multiple dialects. The data from the task and usability survey questions will be used to characterize the subjects’ interactions with the translations systems and to provide feedback from representatives of the target user population to the system developers on how to make these systems more usable. All Likert-scale data will be treated statistically, while the free-form comments will be grouped manually by type.