DARPA BOLT Phase 3 Speech-to-Speech Machine Translation - System User Questionnaire

System User (Subject Matter Expert and Foreign Language Expert) Questionnaire

This statement will be printed on the top of each System User questionnaire.

This collection of information contains Paperwork Reduction Act (PRA) requirements approved by the Office of Management and Budget (OMB). Notwithstanding any other provisions of the law, no person is required to respond to, nor shall any person be subject to a penalty for failure to comply with, a collection of information subject to the requirements of the PRA unless that collection of information displays a currently valid OMB control number. Public reporting burden for this collection is estimated to be 50 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed and completing and reviewing the collection of information. Send comments regarding this burden estimate or any aspect of this collection of information, including suggestions for reducing this burden, to the National Institute of Standards and Technology, Attn: Kay Peterson, MS 8940, 100 Bureau Dr., Gaithersburg, MD 20899.

OMB Control No. 0693-0043

Expiration Date: 10-31-2016

Participants using the translation systems in the evaluation will initiate 5-8 short dialogs with the system solicited clarification option turned ON. After each dialog, participants will answer the following two questions.

Condition: System-Solicited Clarification – ON Scenarios as Initiator (5 minimum; 8 maximum)

Instructions: Please rate the following statements.

Dialog ID	Question	1 Strongly Disagree	2 Disagree	3 Slightly Disagree	4 Neutral	5 Slightly Agree	6 Agree	7 Strongly Agree	N/A
(example) I_S01F01_CLA_T0001_UT000	I achieved my scenario goal.
	The clarification(s) helped during the dialog.

Participants using the translation systems in the evaluation will respond to an additional 5-8 short dialogs with the system solicited clarification option turned ON. After each dialog, participants will answer the following two questions.

Condition: System-Solicited Clarification – ON Scenarios as Respondent (5 minimum; 8 maximum)

Instructions: Please rate the following statements.

Dialog ID	Question	1 Strongly Disagree	2 Disagree	3 Slightly Disagree	4 Neutral	5 Slightly Agree	6 Agree	7 Strongly Agree	N/A
(example) I_F01S01_CLA_T0002_XY012	I was able to communicate successfully.
	The clarification(s) helped during the dialog.

After the completion of the previous 10-16 dialogs with the system solicited clarification option turned ON, participants will answer the following question.

Post-Condition Question

Instructions: Please rate the following statement.

Condition ID	Question	1 Strongly Disagree	2 Disagree	3 Slightly Disagree	4 Neutral	5 Slightly Agree	6 Agree	7 Strongly Agree
(example) I_S01F01_CLA	I felt the system was usable.

Participants using the translation systems in the evaluation will initiate 5-8 short dialogs with the system solicited clarification option turned OFF. After each dialog, participants will answer the following question.

Condition: System-Solicited Clarification – OFF Scenarios as Initiator: (5 minimum; 8 maximum)

Instructions: Please rate the following statements.

Dialog ID	Question	1 Strongly Disagree	2 Disagree	3 Slightly Disagree	4 Neutral	5 Slightly Agree	6 Agree	7 Strongly Agree
(example) I_S01F01_NOC_T0001_UT000	I achieved my scenario goal.

Participants using the translation systems in the evaluation will respond to an additional 5-8 short dialogs with the system solicited clarification option turned OFF. After each dialog, participants will answer the following question.

Condition: System-Solicited Clarification – OFF Scenarios as Respondent: (5 minimum; 8 maximum)

Instructions: Please rate the following statements.

Dialog ID	Question	1 Strongly Disagree	2 Disagree	3 Slightly Disagree	4 Neutral	5 Slightly Agree	6 Agree	7 Strongly Agree
(example) I_F01S01_NOC_T0002_UT000	I was able to communicate successfully.

After the completion of the previous 10-16 dialogs with the system solicited clarification option turned OFF, participants will answer the following question.

Post-Condition Question

Instructions: Please rate the following statement.

Condition ID	Question	1 Strongly Disagree	2 Disagree	3 Slightly Disagree	4 Neutral	5 Slightly Agree	6 Agree	7 Strongly Agree
(example) I_S01F01_NOC	I felt the system was usable.

After the completion of the previous tasks with and without the use of the system solicited clarification option, participants will answer the following questions.

Post-Session Question

Instructions: Please rate the following statement.

Session ID	Question	1 Strongly Disagree	2 Disagree	3 Slightly Disagree	4 Neutral	5 Slightly Agree	6 Agree	7 Strongly Agree	N/A
(example) I_S01F01	I prefer the system with clarification.

Do you have any other comments or feedback? Please describe them below (Do not include any personally identifiable information):

________________________________________________________________________________________________________________

________________________________________________________________________________________________________________

________________________________________________________________________________________________________________

________________________________________________________________________________________________________________

________________________________________________________________________________________________________________

________________________________________________________________________________________________________________