Usability Testing Plan (Best Practices)

Appendix C. Best Practices for Client Participation in Qualitative Data Collection

Abt’s IRB provided the following list identifying best practices for client observations of virtual qualitative data collection in December 2020.

• Use a two-stage recruitment process: A participant will first consent to participate, then Abt will ask a follow-up question to ask if the participant is comfortable having a silent observer from CFPB (similar to consenting to having a resident or medical student observe a procedure). This will let participants have autonomy and choice and also give us and CFPB information about whether having observers is an impediment to any participants.

  • The recruitment language and client introduction should include an explicit statement about why the government client will be observing (e.g., for quality control purposes? Because they need feedback in real time? Because it’s helpful for them to hear this firsthand from program participants?) as well as reinforcing they will only use what they hear for research or program improvement, not for auditing purposes.  

• Limit observers to one per data collection event to reduce the “fishbowl” feeling for respondents.

• Ensure the observer arrives on time for the purpose of consent/disclosure of observation and to eliminate possible disruption.

• Depending on the platform and project, consider asking clients to introduce themselves briefly to show a friendly face, then go off camera and on mute for the remainder of the session.

• Clients should stay in silent/invisible mode throughout data collection. If clients have questions, pass them to Abt (e.g., via a chat window) to pose at the end of a session.

• For privacy protection, no personally identifiable information should be collected. Any notes should be free from identifying details.

• Anything shared during data collection should only be used for the purposes of the contract and not for any other purposes, such as initiating audits or any investigations into compliance, etc.