Study Plan

IOE 2015 BCase01

Experiment 23 Study Plan: Question Grouping

Draft: March 24, 2017

Last revision: March 24, 2017

Version 2.0

* No PII

Principal Investigator: Chris Antoun

Co-PI: Erica Olmsted-Hawala

IOE 2015 BCase 01 Project Team

U.S. Census Bureau

1. Purpose

To investigate the effect of different design approaches for grouping questions on a page in mobile Web surveys.

2. Primary Research Questions

1. What design approach for grouping questions on a page in a mobile Web survey improves respondent performance?

3. Outcome Measures (dependent variables)

Collected by researcher

1. Respondent ratings: a) ease of completion; b) ease of navigation; c) subjective evaluation of completion time; and d) reports of technical problems.

2. Respondent preferences: self-reported preference after seeing all designs.

Collected passively by app

3. Overall completion times: time from selection of the “Begin Survey” button to selection of “Submit Survey” button after the final question.

4. Item missings: the number of questions not answered

5. Edits: the number of answer changes

6. Screen taps: the number of screen taps

7. Answers: responses to the survey questions

4. Experimental Factors (independent variables)

We will vary the design approach (paging vs. scrolling), partitioning scheme (topic-based or not), and loading scheme (lazy loading or not) for grouping items across four different conditions:

1. paging: each question on its own short page

2. scrolling: all questions are displayed on one long page

3. scrolling, lazy-loading: the first question loads by default, once a respondent answers it, the second question loads underneath it. After the respondent answers the second question, the third question loads underneath it, and so on with the page height continuing to lengthen. A “(Show next question)” link just below the most recently loaded question on a page will allows respondents to skip the current question. This link also allows them to move to the next question after answering either an open-ended question or multiple-choice question.

5. Experimental Design

This will be a within-subjects design. Participants will complete three short questionnaires (1-3), one for each condition (A-C). The questionnaires (1-3) and the conditions (A-C) will be counterbalanced across participants (using a balanced Latin Squares approach) in order to distribute possible learning effects. (This approach generates 18 possible orders of questionnaires and conditions.)

Therefore, we are requesting three surveys. Each questionnaire should be programmed in three different ways (one version for each experimental condition). The app should allow the TA to choose between the three questionnaires (1-3) and three conditions (A-C).

6. Experimental Paradigms (tasks)

Participants will be asked to complete a short survey on an iPhone and rate their experience. They will do this three times.

7. Experimental Procedure

For each of the three surveys, the TA does the following: loads the survey app, enters a participant ID, selects an experimental condition, and hands the iPhone to the participant and instructs them to complete the survey. See Appendix A for a detailed data collection protocol.

8. Stimulus Design

Each questionnaire has 9 questions.The items were adapted from the World Values Survey. They are shown in Appendix B. Example screen designs are shown in Appendix C.

9. Data Set

A data set should be produced that has one row for each questionnaire that a participant completes (that is, 3 rows per participant) and 16 columns. The structure is shown in Appendix D.

10. Data Analysis

For each individual questionnaire, we will do comparison across conditions (F test) and then run pairwise comparisons between the conditions. See an example table in Appendix E.

We will then combine all questionnaires. We will fit a linear mixed model using

condition (A-C) and questionnaire (1-3) as covariates along with a random effect of respondent to predict each outcome measure. We may add a variable for the condition order.

11. Participants

The intended sample size is 36 participants (2 participants for each possible order of questionnaires and conditions). Since each participant completes 3 questionnaires, this results in 108 (3 × 36) completed questionnaires.

They should be age 18 and older and own a smartphone. Ideally, all age groups (e.g., 18-34; 35-54; 55+) will be represented so that age can be used as a covariate in the analysis.

Appendix A

Study Protocol

We would now like you to answer three short surveys on a smartphone. In a moment, I will hand you the phone. Each survey should take about 3 to 4 minutes to complete. As a reminder, these questions are for research purposes only and your name will not be associated with your answers.  

You will be timed. You will be timed. So when you take the survey, I want you to imagine that they are doing the survey in a private setting, such as a room by yourself, where you would not be interrupted and where you cannot ask anyone for help. In other words, please do the best you can to complete the survey without taking any breaks or asking for help.

Do you have any questions before we begin?

Ok, great. Now let me load the first survey.

Step 1: [Pick up phone and unlock. Load the Exp 23 app. Bring up the questionnaire (1, 2, or 3) determined from the assignment sheet by tapping the radio button for the desired questionnaire. Bring up one of the 3 designs (A, B, or C) on the smartphone determined from the assignment sheet by tapping the radio button for the desired condition.]

Step 2: [Enter participant ID into the Participant ID field (but DO NOT tap the BEGIN button) before handing device to the P. Hand device to P.]

When the survey is complete, you will see a THANK YOU screen. When you see that screen, please let me know.

Are you ready? Great, here is the smartphone. Please tap the BEGIN button.

Thank you. We have a few questions for you about your experience with this survey.

Step 3: [Take the smartphone back and hand post-survey questionnaire to P]

Ok, great. Now let me load the second survey.

Step 4: [Pick up phone and unlock. Load the Exp 23 app. Bring up the questionnaire (1, 2, or 3) determined from the assignment sheet by tapping the radio button for the desired questionnaire. Bring up one of the 3 designs (A, B, or C) on the smartphone determined from the assignment sheet by tapping the radio button for the desired condition.]

Step 5: [Enter participant ID into the Participant ID field (but DO NOT tap the BEGIN button) before handing device to the P. Hand device to P.]

As before, when the survey is complete, you will see a THANK YOU screen. When you see that screen, please let me know.

Are you ready? Great, here is the smartphone. Please tap the BEGIN button.

Thank you. Once again we have a few questions for you about your experience with this survey.

Step 6: [Take the smartphone back and hand post-survey questionnaire to P]

Ok, great. Now let me load the final survey.

Step 7: [Pick up phone and unlock. Load the Exp 23 app. Bring up the questionnaire (1, 2, or 3) determined from the assignment sheet by tapping the radio button for the desired questionnaire. Bring up one of the 3 designs (A, B, or C) on the smartphone determined from the assignment sheet by tapping the radio button for the desired condition.]

Step 8: [Enter participant ID into the Participant ID field (but DO NOT tap the BEGIN button) before handing device to the P. Hand device to P.]

As before, when the survey is complete, you will see a THANK YOU screen. When you see that screen, please let me know.

Are you ready? Great, here is the smartphone. Please tap the BEGIN button.

Thank you. Once again we have a few questions for you about your experience with this survey.

Step 9: [Take the smartphone back and hand post-survey questionnaire to P]

Great. I have one final question for you.

Step 10: [Hand preference questionnaire to P].

Thank you. You are all done with this experiment.

Appendix B

Requests for programmer:

1. Response options should be vertically aligned.

2. All questions are choose-one questions and should use radio buttons.

3. Radio button icons should be the same size as Exp 18-A (6 mm with 1 mm padding and with clickable text area). Spacing between response options should also follow the same parameters as Exp 18, if possible.

4. After the final question, an Exit screen should appear with this text: “Thank you for your time. Please hand the phone back to the test administrator.”

Italicized text is for reference and is not to be programmed.

Questionnaire 1

Trust in people by group; 5 categories (2 questions)

To what extent do you trust people from various groups? Please indicate your level of trust for each of the following groups on a scale from 1 = no trust at all through 5 = complete trust.

Q1. Your family

Q2. Your neighborhood

Response options: 1 no trust at all, 2, 3, 4, 5 complete trust

Confidence in organization; 10 categories (3 questions)

Can you indicate on a scale from 1 to 10 how much confidence you have in each of the following organizations? 1 means that you have no confidence at all in an organization and 10 means that you have full confidence.

Q3. The press

Q4. The police

Q5. The courts

Response options: 1 no confidence at all, 2, 3, 4, 5, 6, 7, 8, 9, 10 full confidence

Importance; choose-one, 5 categories (2 questions)

For each of the following, indicate how important it is in your life. 1 means not at all important and 5 means very important.

Q6. Family

Q7. Friends

Response options: 1 not at all important, 2, 3, 4, 5 very important

Attitude about science and technology; 10 categories (2 questions)

Now we have some questions about science. Please indicate your level of agreement with each of the following statements on a scale from 1 = strongly disagree through 10 = strongly agree.

Q8. Science and technology are making our lives healthier, easier, and more comfortable

Q9. Because of science and technology, there will be more opportunities for the next generation.

Response options: 1 strongly disagree, 2, 3, 4, 5, 6, 7, 8, 9, 10 strongly agree

Questionnaire 2

Trust in people by group; 5 categories (2 questions)

To what extent do you trust people from various groups? Please indicate your level of trust for each of the following groups on a scale from 1 = no trust at all through 5 = complete trust.

Q1. People you know personally

Q2. People you meet for the first time

Response options: 1 no trust at all, 2, 3, 4, 5 complete trust

Confidence in organization; 10 categories (3 questions)

Can you indicate on a scale from 1 to 10 how much confidence you have in each of the following organizations? 1 means that you have no confidence at all in an organization and 10 means that you have full confidence.

Q3. Political parties

Q4. Universities

Q5. Major companies

Response options: 1 no confidence at all, 2, 3, 4, 5, 6, 7, 8, 9, 10 full confidence

Importance; 5 categories (2 questions)

For each of the following, indicate how important it is in your life. 1 means not at all important and 5 means very important.

Q6. Leisure time

Q7. Politics

Response options: 1 not at all important, 2, 3, 4, 5 very important

Attitude about science and technology; 10 categories (2 questions)

Now we have some questions about science. Please indicate your level of agreement with each of the following statements on a scale from 1 = strongly disagree through 10 = strongly agree.

Q8. The world is better off because of science and technology.

Q9. We depend too much on science and not enough on faith.

Response options: 1 strongly disagree, 2, 3, 4, 5, 6, 7, 8, 9, 10 strongly agree

Questionnaire 3

Trust in people by group; 5 categories (2 questions)

To what extent do you trust people from various groups? Please indicate your level of trust for each of the following groups on a scale from 1 = no trust at all through 5 = complete trust.

Q1. People of another religion

Q2. People of another nationality

Response options: 1 no trust at all, 2, 3, 4, 5 complete trust

Confidence in organization; 10 categories (2 questions)

Can you indicate on a scale from 1 to 10 how much confidence you have in each of the following organizations? 1 means that you have no confidence at all in an organization and 10 means that you have full confidence.

Q3. Banks

Q4. Environmental organizations

Q5. Charitable or humanitarian organizations

Response options: 1 no confidence at all, 2, 3, 4, 5, 6, 7, 8, 9, 10 full confidence

Importance; choose-one, 5 categories (2 questions)

For each of the following, indicate how important it is in your life. 1 means not at all important and 5 means very important.

Q6. Work

Q7. Religion

Response options: 1 not at all important, 2, 3, 4, 5 very important

Attitude about science and technology; 10 categories (2 questions)

Now we have some questions about science. Please indicate your level of agreement with each of the following statements on a scale from 1 = strongly disagree through 10 = strongly agree.

Q8. It is not important for me to know about science in my daily life.

Q9. One of the bad effects of science is that it breaks down people’s ideas of right and wrong.

Response options: 1 strongly disagree, 2, 3, 4, 5, 6, 7, 8, 9, 10 strongly agree

Appendix C

Example screen designs of the first page for each of the four conditions.

A – Paging

To what extent do you trust people from various groups? Please indicate your level of trust for each of the following groups on a scale from 1 = no trust at all through 5 = complete trust.

People of another religion

 1 = no trust at all

 2

 3

 4

 5 = complete trust

Previous Next .

B – Scrolling

To what extent do you trust people from various groups? Please indicate your level of trust for each of the following groups on a scale from 1 = no trust at all through 5 = complete trust.

People of another religion

 1 = no trust at all

 2

 3

 4

 5 = complete trust

People of another nationality

 1 = no trust at all

 2

 3

 4

 5 = complete trust

Previous Next .

C – scrolling, lazy-loading

To what extent do you trust people from various groups? Please indicate your level of trust for each of the following groups on a scale from 1 = no trust at all through 5 = complete trust.

People of another religion

 1 = no trust at all

 2

 3

 4

 5 = complete trust

(Show next question)

After answering question or clicking link…

People of another nationality

 1 = no trust at all

 2

 3

 4

 5 = complete trust

(Show next question)

Appendix D

Data Structure

Definition	Column Number	Variable Names
Participants identification number entered at the start of the session	1	P_ID
Questionnaire number (1, 2, or 3)	2	Questionnaire
Experimental condition (A, B, or C)	3	Condition
Overall completion time	4	CompleteTime
Number of unanswered questions	5	ItemMissings
Number of screen taps	6	Taps
Number of answer changes	7	AnswerChanges
Answers to each question (where 1=the first response option, 2=the second response option, and so forth)	8-16	Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8, Q9

Note: There will be three rows per participant, one for each questionnaire that they completed.

Appendix E

Example results table

Table 1. Completion times (minutes), item missing rate, and respondent ratings with statistically significant differences by group.

	Paging	Scrolling	Scrolling, lazy-loading	F
	(1)	(2)	(3)
Mean completion time	10.2	10.2	10.2	0.98
Item missing rate	10.2	10.2	10.2	0.98
Answer changes	10.2	10.2	10.2	0.98
Screen taps	10.2	10.2	10.2	0.98
Ease of completion	10.2	10.2	10.2	0.98
Ease of navigation	10.2	10.2	10.2	0.98
Subjective completion time	10.2	10.2	10.2	0.98

^* p<.05, ^** p<.01

Then report the format that was preferred by respondents.

Appendix F

Post-survey paper questionnaire. Each participant will fill this out three times. Header will have a place where the TA can indicate the participant number and experimental condition.

How easy or difficult was it to complete this survey?

 1 = Very Easy

 2

 3

 4

 5 = Very Difficult

Think about moving from one question to the next. After answering to a survey question, how easy or difficult was it to move to the next question?

 1 = Very Easy

 2

 3

 4

 5 = Very Difficult

Did you have any technical problems while completing the survey?

 No problems

 Could not select the answer (response not immediately clickable)

 Could not always find the next question to answer

 Some problems while trying to skip a question that I did not what to answer (if applicable)

 Some problems while trying to go back and change a previous answer (if applicable)

 Other, please specify:

Please estimate how much time you took to fill out the questionnaire? (in minutes)

minutes

Post-experiment paper questionnaire. Each participant will fill this out one times. Header will have a place where the TA can indicate the participant number.

Here are four ways to present questions.

Which one do you prefer?

There is no right or wrong answer – we are just interested in your opinion.

(A) (B) (C) (D)

   

*Will replace screenshots from Exp 18 with screenshots from Exp 23.

Why do you prefer that one over the others?

Do your best to briefly explain your thinking: