Supporting Statement Part B -- Understanding Dev Methods from Other Ind to Improve HIT 7-28-2011

SUPPORTING STATEMENT

Part B

Understanding Development Methods from Other Industries to Improve the Design of Consumer Health IT

Version: July 28, 2011

Agency for Healthcare Research and Quality (AHRQ)

Table of Contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent Universe and Sampling Methods 3

2. Information Collection Procedures 4

3. Methods to Maximize Response Rates 4

4. Tests of Procedures 4

5. Statistical Consultants 6

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The proposed qualitative project will use a purposive sample, therefore study findings cannot be statistically generalized to the respondent universe. However, findings will be relevant to inform AHRQ of how the product development process is conceptualized and implemented in other industries, thus informing potential strategies for the development of products related to consumer health information technology (IT).

The project team conducted an initial activity– an environmental scan and grey literature review – in order to identify consumer products that are relevant to the development of consumer health IT products. The project team identified products that fit one or more specific criteria, including but not limited to:

(1) Type of role that the product plays in users’ lives;

(2) The product’s capability to accommodate a wide range of individual differences in use patterns and information needs;

(3) The product’s capability for long-term monitoring of information;

(4) The capability of the product to centralize and store information from multiple sources; and

(5) The capability of the product to accommodate data of multiple types and varying levels of complexity.

The preliminary results of this activity yielded the following seven main categories of consumer products that will receive in-depth investigation:

1. Communication (including social networking);

2. eCommerce;

3. Information storage, archiving, and retrieval;

4. Personalized entertainment;

5. Gaming;

6. Learning applications; and

7. Smart phones¹.

Within each of the seven categories, the project team identified numerous specific successful products across industry types that have been marketed to, and adopted by, consumers. The criteria for defining a “successful” consumer product have been defined within this project as: market penetration (based on number of users), sales revenue, accolades in design press, and user adoption and enjoyment (based on positive product reviews).

The original list of successful consumer products was supplemented with selections from periodicals such as Consumer Reports, PC Magazine’s best and most successful products of the year and product design organizations that grant awards for consumer products such as the Industrial Designers Society of America.

The selected products have a profile of features or functions that will be relevant for consumer health IT, or use patterns similar to, or desirable for, consumer health IT products. The team identified products that support the following types of activities:

• Storing, archiving, and retrieving information;

• Monitoring health-related information;

• Searching for information; and

• Tools that support and ease completion of tasks such as logging/recording activities, making comparisons, and making decisions.

Additional types of successful product groups were included in areas such as communication, personalized entertainment, and gaming as these were considered product areas that received a great deal of growth and success in the last few years. From a list of around 250, products that were most successful or relevant to consumer health IT were selected for further exploration. After finalizing criteria for selecting 15 products of most interest, the team developed a list of 15 potential key informants associated with these identified products. A list of alternate informants has also been developed in case the primary informants are not available for interviews.

Study participants will be identified to represent these industries and products. The total of 15 in-depth qualitative interviews planned for the interview phase was based on representation of approximately two consumer products within each category, to allow for variation across industries.

There are no strict criteria for sample size. However, qualitative studies of this nature typically include 10-20 participants because including fewer participants limits researchers in their ability to make sound recommendations (Patton, 2001; Creswell, 2007). Qualitative studies typically employ multiple forms of evidence and there is no statistical test to determine significance of results when conducting qualitative research. The researcher determines usefulness and credibility of results through the process of inductive analysis of data.

The project team will develop and finalize criteria for selecting three products of primary interest from each consumer product category. Then the project team will identify the list of potential informants (henceforth called study participants) associated with these identified products.

2. Information Collection Procedures

Key Informant Recruitment. The project team will contact the potential study participants directly or approach the public relations (PR) office associated with each product (e.g., through contact information provided on organizations’ Web sites).

Attachment C provides a sample invitation letter. The purpose of the communication is to explain the intent, gain the organization’s buy-in, determine an appropriate informant, and describe the commitment. Each organizational informant who agrees to participate in the study will receive a written confirmation of the interview date and time by e-mail or fax (see Attachment D).

3. Methods to Maximize Response Rates

Technical experts are not being selected via probability-based sampling methods. A “response rate” has no clear meaning in the context of this study. A list of “alternate” study participants will be created in case any targeted study participants decline to participate.

4. Tests of Procedures

The interview protocol was developed by drafting questions based on commonly accepted product development phases of: (1) idea generation, (2) concept development, (3) iterative testing, (4) implementation, and (5) commercialization. Specific probes reflect this study’s focus on particular products and their features that are relevant for consumer health IT.

The interview protocol was reviewed by the Project Director Enid Montague, Ph.D., Assistant Professor and Anna Julia Cooper Fellow, as well as Pascale Carayon, Ph.D., Professor, both of the Department of Industrial and Systems Engineering, University of Wisconsin-Madison. Drs. Montague and Carayon have substantial experience in developing and fielding interview protocols and surveys for product development professionals in a variety of industries.

AHRQ has not identified any prior qualitative studies investigating industry design processes. Therefore, the interview guide received additional review and testing.

First, four individuals were asked to review the interview guide. The review process was based on accepted methods for reviewing an interview guide for comprehensiveness, clarity of wording, and any issues related to bias and communication (Wolfe & Smith, 2007). One layperson was selected to review the questions with a focus on clarity of wording. Three of the individuals were selected based on their expertise in one of the following industries: communications, health care, and design technology.

Based on the feedback received from the review of the interview guide, it was revised as described below:

• Question Q.C.01.d.in section Q.C “Concept Development” of the interview guide was revised from “Does your firm use any contemporary approaches to product development?” to “Does your firm use any contemporary or modern approaches to product development?” to ensure that the meaning of the term “contemporary approaches” is easily understood by informants.

Second, a pilot interview was conducted with a product developer. This participant was selected to closely represent the target population for the key informant interviews.

Based on the pilot interview results the following revisions were made to the interview guide:

• A new section called “General Company Information” (now section Q.A of the interview guide) was added to the beginning of the interview guide to account for informants who may not be involved in all stages of product development. This new section includes general background questions about the size of the company, product focus, product development phases, and individual team member involvement.

The questions listed below were added to section Q.D “Testing” of the interview guide to better understand how each firm distinguishes between usefulness and ease of use of a product. These concepts are critical to design, but often combined even though they are distinct concepts:

Q.D.02. How does your firm define usefulness versus ease of use of a product?

Q.D.02.a How does your firm gauge the perceived usefulness of a product?

Q.D.02.a.1 When does this typically occur?

Q.D.02.b How does your firm gauge the perceived ease of use of a product?

Q.D.02.b.1 When does this typically occur?

Finally, based on feedback from both interview guide reviewers and the pilot interview participant the following changes were made:

• The question “When does [the product development phased being discussed] typically occur?” has been added to identify when each of the development phases discussed in the interview guide takes place. This question was added because the reviewers and the pilot interview participant noted that product development phases may ensue at various times for different companies. These phases and new question numbers are listed below:

• Section Q.B: Idea Generation

  • Initiation of product development processes (Q.B.01.a)

  • Identification of end users (Q.B.01.c)

• Section Q.C: Concept Development

  • Prototyping (Q.C.01.c)

• Section Q.D: Testing (Q.D.01.a)

  • Usefulness of a product (Q.D.02.a.1)

  • Ease of use of a product (Q.D.02.b.1)

• The following two questions were moved from section Q.D “Testing” to section Q.C “Concept Development” based on feedback from the reviewers and pilot interview participant that segments of the consumer market are identified earlier in the product development cycle – when the product characteristics are defined:

Q.C.02. In what ways does your firm segment the consumer market?

Q.C.02.a. What role do these segments play in tailoring the testing of a product?

Following the review and testing process, a new section: “Q.G. Individual’s Experience With and Perspective on Design Methods Applicable to Consumer Health IT” was added. Based pilot interview results, we observed that the participant’s company worked with health and wellness IT products and was interested in transferring methods in their core domains to the field of consumer health IT. Therefore, section Q.G was added to better understand the positive and negative experiences that companies have with using existing design methods in the consumer health IT domain.

5. Statistical Consultants

Because this effort is focused on obtaining technical consultation from industry experts, no statistical consultants were contacted.

The project director, Dr. Enid Montague, is a consultant to Westat, the prime contractor. She will be responsible for overseeing the activities related to this work, including identifying participants, training the interviewer who will conduct all of the interviews, planning the qualitative analysis steps and reporting the findings. Dr. Montague can be reached by phone at (608) 890-3546 or by email at [email protected] project manager from the contractor organization, Westat, is Jennifer Crafts, Ph.D. She can be reached by phone at (301) 610-4881 or by email at [email protected].

References

Patton, M. Q. (2001). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: SAGE.

Creswell, J. W. (2007). Qualitative inquiry and research design (2nd ed.). Thousand Oaks, CA: SAGE.

Wolfe, E.W. & Smith, E.V. (2007). “Instrument development tools and activities for measure validation using rasch models: Part I development activities.” In Smith, E.V. & Smith, R.M. eds. Rasch measurement: Advanced and specialized applications. Maple Grove, MN: JAM Press.

1 A smart phone is a mobile phone that offers more advanced computing ability and connectivity than a typical mobile phone and may include, in addition to digital voice, text messaging, e-mail, Web browsing, still and video cameras, music player, and video viewing. Smart phones can run multiple applications, turning a cellular phone into a mobile computer.