Lori McLeod, PhD and Sheri Fehnel, PhD

This workshop is intended to be an introduction to psychometrics and its value in outcomes research. Our goal is to equip participants with a basic understanding of commonly used psychometric techniques and criteria appropriate for evaluating the utility of instruments used to measure patient-reported outcomes, including quality of life.
We will provide participants with copies of our presentation, including recommended references for further information about instruments, techniques, and software, as well as a handout with relevant computational formulas.
This workshop is intended to be an introduction to psychometrics and its value in outcomes research. The purpose and general principles behind the most commonly used psychometric techniques such as exploratory factor analysis and structural equations modeling, as well as various approaches to item response theory (e.g., Rasch, 2-parameter logistic and graded-response models) will be described. In addition, for each psychometric property such as reliability, validity, and responsiveness, a bit of history, definitions, estimation methods, and criteria appropriate for evaluating the utility of outcomes instruments will be provided. Real-life examples will be used to illustrate what tradeoffs among various instrument attributes may be appropriate given specific research goals. Additional issues that will be discussed include the interpretation of patient-reported outcomes data (e.g., defining minimal clinically important differences), computer-adaptive testing, and when the development of a new instrument may be preferable to the use of an existing instrument. All throughout the workshop, the presenters will entertain questions from the workshop participants about the development of questionnaires to measure patient-reported outcomes, psychometric methods, and the selection of outcomes instruments.
This is a basic level workshop.

Chih-Hung Chang, PhD and Bryce Reeve, PhD

Although item response theory (IRT) models have been developed and widely used in educational and psychological testing for many decades, their use in health-related quality of life (HRQOL) measurement has just flourished. Theoretically-sound IRT models coupled with available software packages make it possible for researchers and practitioners to develop and refine HRQOL assessment instruments for use in research and clinical trials. This workshop will provide an overview of IRT models and how they can be appropriately applied to HRQOL assessment. Specifically, the following topics will be discussed: 1) dichtomous vs. polytomous models; 2) unidimensional vs. multidimensional models; 3) instrument/scale construction; 4) exploratory vs. confirmatory item-level factor analysis; 5) instrument equating; 6) differential item functioning; 7) item banking and computerized adaptive testing; and 8) software availability and capacity. Examples using empirical data and annotated computer outputs will be provided and discussed. Guidelines to the selection of models and software will also be provided.
This is a basic level workshop.

INTRODUCTION TO QUALITATIVE AND QUANTITATIVE METHODS FOR ASSESSING MEASUREMENT EQUIVALENCE IN DIFFERENT SUBGROUPS

Ron Hays, PhD and Leo Morales, MD

A fundamental underlying assumption in comparative studies among racial/ethnic and linguistic subgroups is that the measures reflect actual differences in the health outcomes of interest rather than differences attributable to measurement bias (Stewart & Nápoles-Springer, 2001). This workshop will discuss the use of qualitative (focus groups, cognitive interviews) and quantitative (classical test theory and item response theory) methods for evaluating the equivalence of health-related quality of life measures between different subgroups. Focus groups can also be used to diagnose potential problems with a new survey instrument and to assess the cultural appropriateness of a survey instrument. For example, focus groups can be used to assess whether a survey adequately covers the intended domains in a target group. They can be used to assess if the concepts used in a survey are emic or etic to the target group. Finally, they can be used to assess if specific terms (or translated terms) make sense to the target group. The key steps in preparing for a focus group are to formulate a research questions; identify a sampling frame; identify a moderator; develop the protocol; recruit the sample; conduct the focus group(s); and analyze and summarize the data (Stewart & Shamdasani, 1990). Cognitive interviewing techniques can be used to identify a variety of problems including confusing or poorly constructed instructions on how to complete the survey; confusing words, terms or jargon; questions that are being interpreted inconsistently or not as they are intended; errors in survey formatting such as confusing or incorrect skip patterns in the survey instrument; respondent difficulties in knowing or recalling information or events asked about in the survey; ambiguity in the questions or response category options; ambiguity in the questions or response category options; inadequate fit between the question and the response options; and redundancy in content across multiple questions. Cognitive interviews can identify emic and etic concepts in a survey, when answers are compared across cultural or linguistic groups. Cognitive interviews can also identify problematic translations. Finally, cognitive interviews can be used in estimating the readability level of a survey (Morales et al., 2001).
This is an intermediate level workshop.

Ivan Barofsky, PhD and Elaine McColl, PhD

This is an intermediate level Workshop designed to demonstrate the relevance of the cognitive sciences to HRQOL research. A participant will be introduced to cognitive interviewing as one method to study the role of cognition in HRQOL assessments. Participants will also be asked to conduct a cognitive interview following administration of each of the four items on the SF-36 dealing with general health perception. Particular attention will be paid to identifying the presence of cognitive mechanisms (e.g., heuristics) in response to the rating task. A subsequent discussion will illustrate data analysis procedures and issues following cognitive interviewing. Results from experimental studies dealing with cognitive issues (e.g., recall of medical events, context effects, and so on) will also be reviewed.
This is a basic level workshop.

Sarah Conner, MS

The appropriate and efficient use of limited health resources is a perennial issue among health-care providers, policy-makers, and society at large. Information, and ideally, evidence, must form the basis upon which decisions are made. In the past, the impact (burden) of non-fatal outcomes of disease and injury on population health tended not to receive appropriate policy attention. However, healthy living and quality of life are now increasingly recognized as important policy goals, and research on the burden of disease helps to quantify the impact of non-fatal diseases. This workshop focuses on the main measures for eliciting health state preferences from the general population within the context of a Burden of Disease Study, which provide a means for considering the significance of non-fatal outcomes. This requires eliciting a numerical value or a weight reflecting the population's relative preference for each of a series of diseases or health states.
The main tools available for measuring these health state preferences will be presented and the methods for eliciting preference scores will be briefly described. In order to gain a greater understanding of the advantages and disadvantages of each of the methods, participants will then be taken through a series of exercises that will simulate the panels used for eliciting health state weights. Participants will begin by ranking health states using a Visual Analog Scale (VAS). The VAS is used as a teaching tool to familiarize participants with the health state descriptions and to answer any questions that may arise. A facilitator will then help the group to generate weights using each of the 3 main instruments that have been tested in the Canadian study including the Time Trade-Off, the Standard Gamble and the Person Trade-Off. They will be given the opportunity to discuss their responses and to reach a well thought out conclusion. Results will be fed back to participants so they can see the ratings achieved by each of the methods and discuss the findings.
This is a basic level workshop.

Marcia Testa, PhD, MPH

Health outcomes research involves the scientific inquiry evaluating the results of medical interventions and health care services to determine which interventions and services influence the probability of optimal patient outcomes, including the patient's physiologic status, physical functioning, emotional and intellectual performance and comfort. Both the independent factors representing health care services and the dependent patient outcomes can be multidimensional, multi-layered and measured indirectly. Since the impact of treatment might be direct for some outcomes and indirect for others, information explaining the potential causal pathways among treatments and outcomes becomes critical when designing quality-of-care and quality-of-life improvement strategies. When using unobserved constructs in a model, latent variable structural equation modeling might offer a potential advantage over traditional analysis techniques, such as multiple regression, since it involves combining measurement models and structural equations. First, multiple indicators can be used to measure latent quality-of-life variables such as physical, psychological and social health functioning. Secondly, by using a measurement model, unobservable or observable constructs can be operationalized by connecting them to one or more observed measures. In this workshop, we will review the elements of latent variable structural equation modeling that are useful when patient outcomes are represented by multiple indicators and multiple constructs. Examples from therapeutic clinical trials involving the impact of treatment on clinical factors, quality of life and treatment satisfaction will be presented.
This is an advanced level workshop.

Kathleen Wyrwich, PhD

Although numerous measures have been developed for the evaluation of health related quality of life (HRQoL), strategies for identifying meaningful intra-individual and group change in these measures have not kept pace with instrument development. As a result, clinical trial researchers, quality assurance assessment teams, practicing clinicians, and patients are without established standards to evaluate change in HRQoL measures. This course will review, critique and compare the methods that have been applied to establish intra-individual and group HRQoL change standards, which include anchor- and distribution-based techniques. Practical approaches to improving and advancing HRQoL change evaluations that enhance the interpretation of intra-individual and group change, as well as a review of controversies that have developed will be provided. In addition, the course will explore future qualitative and quantitative challenges in this area of HRQoL research.
This is an advanced level workshop.

John Ware, PhD, Jakob Bjorner, MD, PhD, and Mark Kosinski. MA

This advanced workshop begins with an overview of trends in the standardization of health metrics and, for the most frequently measured concepts, how methods can be matched to the requirements of population surveys, clinical trials and individual patient assessments while maintaining the comparability of results. Applications of item response theory (IRT) focusing on the measurement of generic and disease-specific outcomes will be used to illustrate the overall analytic approach recommended by the faculty. Methods and results from specific analyses and software packages used at each step will be presented to illustrate item trace line exploration (Testgraf), the factor analysis of categorical data to test for multidimensionality (Mplus), Rasch item response model estimation (OPLM), and IRT modeling (Parscale). The "modern" psychometric approach to item testing will be contrasted to the "classical" approach. The logic of computerized adaptive testing (CAT) applied to dynamic health assessment software (DYNHA, QualityMetric) will also be illustrated. The goals are to explain the "why" and "how" of developing the best items and improving estimates of parameters used in scoring items and the advantages of administering them using the logic of CAT. Other advanced topics that will be covered include: (a) estimation of item and test "information functions" at specific scale levels and their implications for estimating score reliability and confidence intervals for individual patients throughout the score range (b) strategies for using IRT models for missing data estimation; (c) cross-calibration of widely-used measures; and (d) demonstrations of the practical implications of CAT for purposes of population screening and monitoring individual patient health outcomes.
This is an advanced level workshop.

William R. Lenderking, PhD and Mark A. Blais

Although the fields of psychology and mental health have contributed greatly to outcomes research, particularly with respect to methodological contributions, most mental health outcomes measures have focused on measuring changes in the symptoms of specific diseases. On the other hand, most generic health status instruments are not specific enough to be optimally sensitive to changes in mental health conditions. This workshop will describe the development of a mental health outcome measure, the Schwartz Outcomes Scale (SOS-10), that is sensitive to changes across a variety of mental health conditions without being a symptom measure. The presenters will use the SOS-10's development process and accumulated research data to illustrate how the combined use of classical test theory and modern test theory, in particular Rasch modeling, can be successfully employed in scale development. Strengths and weaknesses of both classical test theory and modern test theory will be discussed and illustrated as well as the impact these limitations have on measurement and scale quality. Important measurement principles such as dimensionality (uni vs. multi), quality of data (ordinal vs. interval), and response scaling will be reviewed and discussed from both classical and modern test theory perspectives. Participants will gain practical knowledge in how to apply the ideas from these differing traditions to improve their own development of new scales to measure mental health outcomes.
This is an advanced level workshop.

John Brazier, PhD

This workshop is concerned with the use of measures of health related quality of life (HRQoL) in economic evaluation, including their use in deriving the health state utility values as required to calculate Quality Adjusted Quality Life Years (QALYs). The workshop is designed to provide a practical introduction to the problems and issues around the topic. It assumes a basic knowledge HRQoL measurement and preference-elicitation techniques (such as standard gamble and time trade-off). Students unfamiliar with the latter are recommended attending the ISOQOL workshop introducing Health State Preference/Utility Assessment.
There will be four brief presentations:

1. The limitations of using HRQoL measure in economic evaluation
2. Methods for adapting non-preference-based measures for use in economic evaluation, including arbitrary weighting, verbal mapping, estimation of exchange rates and the estimation of preferences from valuation surveys.
3. A review of existing preference-based measures (e.g. HUI-III and EQ-5D)
4. Estimation of preference-based measures of health from the SF-36 and other measures of quality of life

These presentations will use practical example s throughout.
Organization of workshop: 50% will be composed of these four brief presentations on each of these topics, 35% on a small group exercise addressing the appropriateness of different methods for obtaining health state utility values in different contexts and 15% Q&A (though questions are welcomed throughout).
This is an advanced level workshop.

Diane Fairclough, DrPH

Non-ignorable missing data is not uncommon when HRQoL is measured in longitudinal studies where participants may experience morbidity or mortality. This workshop will examine a number of models that are commonly proposed for the analysis of these studies. The models include pattern mixture models, conditional linear model (Wu and Bailey, 1989), joint mixed-effects and time-to-effect model (Schluchter, 1992), and outcome-dependent logistic dropout selection model (Diggle and Kenward, 1994). Specifically, we will examine both the underlying assumptions and the practical constraints for a number of pattern-mixture and selection models. The procedures for implementing each of the models will be presented and illustrated using data from a clinical trial measuring HRQOL in patients with advanced lung cancer.
Participants should have some experience with multivariate analysis of longitudinal data or repeated measures including MANOVA and mixed-effects models.
This is an advanced level workshop.

Alex Michalos, PhD

This workshop will begin with the definition of key terms, including examples. We will then explore possible uses and abuses of measures of subjective well being, and then summarize critical issues in subjective well-being measurement. Facts and theories about subjective well-being, happiness and satisfaction covering the following issues: Alternative measures and their relations, international comparisons, stability and changeability of indicators, what we know about transient moods and mood effects, what we know about confounding influences and nonsampling errors, what we know from meta-analysis abut the relative explanatory power of some demographic variables, what we know about reported job satisfaction , what we know about reported marital satisfaction, different kinds of explanatory theories, relative strength of simple linear model of explanation, top-down, bottom-up and bi-directional models and discrepancy theories and multiple discrepancies theory (MDT).
This is an advanced level workshop.

David Osoba, MD, Andrea Bezjak, MD, MSc, Michael Brundage, MD, and Joseph Lipscomb, PhD for the QOL Committee of the National Cancer Institute of Canada (NCIC) Clinical Trials Group (CTG), Canada, and the National Cancer Institute (NCI), USA.

HRQOL assessment in oncology clinical trials has increased rapidly over the past 15 years. Currently, several cooperative oncology groups are including HRQOL assessment in substantial proportions of their trials. To achieve success, it is important to have clear guidelines about how HRQOL assessment is to be carried out, how the analysis of the data will proceed and how the results will be communicated to patients, health care professionals and policy makers.
In this workshop, we will provide a summary of the procedures followed by one cooperative clinical trials group, the NCIC CTG, to include HRQOL measurement in clinical trials and to achieve high questionnaire completion rates. Next we will present a step-by-step approach to the logical analysis and interpretation of the collected data, with examples that illustrate how HRQOL results have made a difference in the interpretation of the results of clinical trials. This will be followed by an exploration of some of the challenges involved in the translation and presentation of HRQOL data to users of such information, including patients, care-givers and policy-makers. In addition, a perspective on some of these issues from a current project of the USA NCI on cancer outcomes measurement will round out the workshop.
By the end of the presentations, participants should have gained a better understanding of the necessary components and steps for the conduct of successful HRQOL measurement in oncology clinical trials.
This is an advanced level workshop.

Carol M. Moinpour