Attachment 11 - Analysis Plan

Attachment A11 - Analysis Plan_final.docx

Consumer Expenditure Surveys: Quarterly Interview and Diary

Attachment 11 - Analysis Plan

OMB: 1220-0050

Document [docx]
Download: docx | pdf

Gemini Proof of Concept Test

Analysis Plan October 9, 2014


Methodological Issues

Concept

Analysis

Comparison Group

Evaluation Guidelines1

Research Objective 1. Feasibility of completing one wave of the proposed survey redesign

Feasibility of Design


Overall response rate: % HHs that complete recall, complete at least one diary and complete records interview

Production CEQ and CED rates; same counties as in the test

RR must be higher than 60%, or no lower than 10% of production


Amount of data collected: total expenditures reported vs. production (diary + interview)

Production total expenditures, CEQ and CED; same counties as in the test

Total expn. must be no lower than 10% of production


Quality of data collected: Percent interview and diary data that had to be allocated or imputed

Percent production interview and diary data that had to be allocated or imputed

Editing rates must be no higher than 10% of production


Subgroup comparisons: Compare overall response rates, quantity and quality of data across demographic subgroups (gender, age, race, household income, household size)

Production CEQ and CED rates; same counties as in the test

No decision point

Research Objective 2. Effectiveness and costs of incentives

Overall effectiveness

% of HHs that reported receiving advance letter (with $2)

None

More than 50% of HHs say yes


% of debit cards that were cashed

None

No decision point

Gaining cooperation

Overall response rate: % HHs that complete recall, complete at least one diary and complete records interview

Production CEQ and CED rates

Individual Diary Feasibility Test (IDFT) response rates

RR must be higher than 60%, or no lower than 10% of production

Reducing contact attempts

Number of contact attempts prior to Visit 1, as indicated by time stamp of collection of HH Roster

Production number of contact attempts prior to Wave 1

Production average number of contact attempts prior to CED

Number of contacts attempts should be equal or less than to the average number of contact attempts required for wave 1 and diary placement


Average total number of contact attempts

Production average total number of contact attempts for the first and second wave cases + diary cases

Number of contacts attempts should be equal or less than to the average number of contacts in production

Encouraging engagement

% of HHs that received recall incentive, diary incentive, full/partial records incentive

None

More than 60% of HHs were eligible for all three incentives

Research Objective 3. Respondent willingness to complete all components of the integrated survey

Overall Response Rates

Overall response rate: % HHs that complete recall, complete at least one diary and complete records interview

Production CEQ and CED rates (duplicate item)

RR must be higher than 60%, or no lower than 10% of production


% of HHs that refuse one or more survey elements

Production CEQ and CED refusal rates

Refusal rate must be no lower than 10% of production

Recall Interview

% of HHs that refuse the recall interview

Production CEQ refusal rates for Wave 1

Refusal rate must be no lower than 10% of production

Diary

% of eligible HH members who complete diary

IDFT

At least half of eligible HH members complete (1+ entries) a diary; or were eligible blanks

Records Interview

% of HHs that refuse the records interview

Production CEQ refusal rates for Wave 2

Refusal rate must be no lower than 10% of production

Records

Number and type of records provided by each HH

None

At least 75% of participating HHs provide one or more records (as indicated by FR)

Operational Issues

Research Objective 4. Number/Effectiveness of contact attempts necessary to obtain a completed interview

Contact Attempts

Number of contact attempts prior to Visit 1

Number of contact attempts prior to first wave in production, and

Number of contact attempts prior to production CED

Number of contacts attempts should be equal or less than to the average number of contact attempts required for wave 1 and diary placement


Number of mid-week Contacts

IDFT contact rates

None, used to understand FR effort & R response across all parts of interview


Effectiveness of mid-week Contacts in obtaining additional entries

Comparison of number of entries in individual diaries prior to mid-week contact to post mid-week contact; compare individual diaries with no mid-week contact to diaries with mid-week contact

None, used to understand the effectiveness of contacting the respondent during diary week.


Number of contact attempts prior to Visit 2 as indicated by the time stamp for the first records CAPI section

None

None, used to understand FR effort & R response across all parts of interview


Total number of contact attempts

Production total number of contact attempts for the first and second wave cases + diary cases

Total number of contacts attempts should be equal or less than to the average number of contact attempts required for wave 1 and diary placement

Research Objective 5. Length of the Interview Elements

Timing

Average time of recall interview

Average time of “recall sections” in production CEQ

Average recall interview is less than 60 minutes


Average of each Recall Section

Average of “recall sections” in production CEQ

No decision point


Average Visit 1 time


Average visit 1 is less than 60 minutes


Average Visit 2 time, excluding debriefing


Average visit 1 is less than 60 minutes


Average time spent at diary placement screen to determine time it takes to review instructions / length of diary placement


No decision point


Length of Diary Recall Process


No decision point


Total time of records interview

Total time of “records sections” in production CEQ

Average records interview is 60 minutes


Time spent at record placement screen to determine time it takes to review instructions / records placement


No decision point


Time of each Records section

Time of “record sections” in production CEQ; Time of “record sections” in production CEQ for Rs who used records “almost always”

No decision point

Research Objective 6. Technical issues respondents have with completing the online diaries

Reported Issues

Log of calls (and emails) to help desk and reasons for the contact




Respondent comments during debriefing


No decision point

Observed Issues

Number of times Rs accessed the change password page but was not able to change their password


No decision point


Total time R spent on change password page, broken down by successful password change and non successful password change




Number of unsuccessful log-in attempts


No decision point


Number of blank diaries, with and without successful log-ins


At least half of HH who were given a web diary members completed it (1+ entries); or were eligible blanks


Item missingness (e.g. amount, characteristic)*


No decision point


Number of items reported per diary and per HH, by category and overall*

Number of items reported by CU, by category and overall in production CEQ and CED

No decision point


Amount of expenditures reported per diary and per HH, by category and overall*

Amount of expenditure reported by CU, by category and overall in production CEQ and CED

No decision point


Number of receipts provided by HH with more than 5 items* (from FR debriefing)


No decision point


Number of paper diaries completed by Rs given electronic diaries

Compare mode placed variable with the mode of most expenditure entries

No decision point

Non-Reported Issues

R issues identified by FRs through case notes and/or debriefing


No decision point

Research Objective 6a. Individual Diary Placement and Pickup Process

Diary Placement Process

Number of HH members present during placement (FR debriefing)


No decision point


Number of DK responses for proxy placement questions


No decision point


R debriefing questions about diary placement


No decision point


FR debriefing questions about diary placement


No decision point


Whether a login attempt was made during placement (FR debriefing)



Diary Pickup Process

Number of HH members present during pickup


No decision point


Number of DK responses for proxy pickup questions


No decision point


Percent of individual diary keepers who received an incentive


No decision point


FR debriefing questions about diary pickup


No decision point

Research Objective 7. Per completed interview cost of administering the survey


Total number of contacts per case


Total number of contacts attempts should be equal or less than to the average number of contact attempts required for wave 1 and diary placement


Total interviewing time per case


Total interviewing time (Visit 1 + Visit 2) should be less than 180 minutes


Total amount of incentives given per case


No decision point


Total amount of incentives cashed per case


No decision point


Other costs (travel, preparation) as available from Census


No decision point


Total cost (interviewing time + incentives + other costs)


Estimate of total survey costs, excluding incentives, is within 10% of current production budget

Experience

Research Objective 8. Respondent Experience

Respondent Debriefing

Analysis of R debriefing questions


No decision point

Research Objective 9. FR Experience

FR Debriefing

Analysis of FR debriefing questions


No decision point

* Not definitive measure of technical difficulty

1 The proof of concept test goal is to measure the feasibility of implementation of the redesign plan as well as the improvement of measurement. The guidelines are a starting point for evaluating these goals. Looking at all of the guidelines as a whole will help determine the success of the design.

5


File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorEdgar, Jennifer - BLS
File Modified0000-00-00
File Created2021-01-25

© 2024 OMB.report | Privacy Policy