SlideShare a Scribd company logo

[Typ]Presentation[Sbj]LaboratoryDiagnosisDefined[Dte]20131028

•
•
M
Mark Gusack

The document discusses the changing role of laboratories in diagnosis and diagnostic decision making. It notes that historically, diagnoses were labels placed on symptoms with little need for testing or effective treatment. Now, accurate diagnoses have serious consequences given more effective therapies. This has led to increased complexity and expectations in diagnostic medicine. The laboratory now plays a central role in the diagnostic process. Establishing acceptable risk for patients requires understanding how test validity, value, and clinical risk are determined. Factors like test purpose, reliability, and how results are used must be considered to optimize outcomes. Educating clinicians is important so medical decision points factor in test limitations and implications for patient risk.

Related slideshows

Epidemiological study designs
Epidemiological study designsEpidemiological study designs
Epidemiological study designs
 
Criticisms of orthodox medical ethics, importance of
Criticisms of orthodox medical ethics, importance ofCriticisms of orthodox medical ethics, importance of
Criticisms of orthodox medical ethics, importance of
 
Noncommunicable diseases - descriptive & analytical studies
Noncommunicable diseases - descriptive & analytical studiesNoncommunicable diseases - descriptive & analytical studies
Noncommunicable diseases - descriptive & analytical studies
 
1 of 44
Download to read offline
THE LABORATORY’S ROLE IN DIAGNOSIS AND DIAGNOSTIC DECISION MAKING Mark Gusack, M.D., Staff Pathologist VA Medical Center, Huntington, WV Adjunct Clinical Professor Marshall University Medical School NORTHEAST LABORATORY CONFERENCE 15 OCT 2013 RISK
Mark Gusack, M.D., OCD [Overly Concerned Doctor] Staff Pathologist, Pathology and Laboratory Medicine Service VA Medical Center Huntington, WV 25704 mark.gusack2@va.gov SPEAKER Dr. Gusack has almost 40 years experience in the Laboratory field starting as a Nuclear Medicine Technologist in the early 1970’s, then working as a clinical engineer, and then becoming a physician and pathologist. He is AP/CP boarded, has held positions in a variety of hospital and reference based laboratories as a medical director and as staff pathologist. During this time he has also been a consultant and practiced as a Licensed Health Care Risk Manager in Florida. Dr. Gusack has been involved with all aspects of laboratory development and management including startup, licensing, as well as designing integrated management systems for clinical laboratories. The opinions expressed in this presentation are those of the author and do not necessarily represent those of the Department of Veterans Affairs
OUR MISSION HOW DO WE CARRY OUT THIS MISSION? OUR MISION IS TO PROVIDE THE VERY BEST CARE TO OUR PATIENTS
SITUATION – PART I HOW HAS THIS AFFECTED WHAT WE DO IN THE LABORATORY? The meaning of diagnosis has changed radically over the past 60 – 70 years. THEN:  In the past, there were few diagnostic tests and fewer effective treatments.  A diagnosis was a label placed on a set of signs and symptoms with a prognosis.  Since most therapy was ineffective error in diagnosis was rarely a critical issue. NOW:  Since that time our fund of knowledge has grown prodigiously.  Our ability to apply this knowledge to health care has transformed medicine.  An error in diagnosis has serious consequences given more effective therapy. THE CONSEQUENCES:  The patient can be harmed or die due to a false positive or negative diagnosis.  Increased morbidity and mortality in a population can cost a society greatly.  Increased numbers of failure in the face of unprecedented success.
SITUATION – PART II SO WE HAVE INITIATED A CYCLE OF:  Increased subdivision of older diagnoses into numerous newer diseases.  Increase capacity to make an accurate diagnoses.  Increased ability to effectively treat more diseases. LEADING TO:  A dramatic increase in the complexity of diagnostic medicine.  A dramatic increase in the perceived impact of a diagnosis. CONSEQUENCES:  Increased expectations for a “perfect” outcome.  Increased apparent error in diagnosis.  Increased perception of harm by the patient and society. Now standards of Acceptable Risk are much more stringent
SITUATION – PART III – THE RISE OF CHRONIC DISEASE THIS HAS DRAMATICALLY CHANGED THE MEDICAL PROFESSION Today most acute diseases have been eliminated through: Civil Engineering: Waste, water, and food safety management Improved Diet: Improving our capacity to survive an illness Modern Medicine: Improved therapies to cure acute illness As a result today we are dealing primarily with:  Chronic, progressive, debilitating diseases for which  There is often no definitive curative therapy which  Cost far more to treat than acute illnesses which  Escalate over time as the patient progressively declines in health Present national budget problems are due,to a great degree to this progression And what WE do will determine, to a great degree the final outcome.
PROBLEM How can we reduce the impact of Systematic Complexity of the Diagnostic Process on how we generate test results in order to assure establishing: Acceptable Risk For our Patients?
SOLUTION PART I THIS IS NOT GOING TO BE EASY In order to establish Acceptable Risk we need to:  Define what the term diagnosis means  Understand the difference between the concept of a diagnosis and the process of making a diagnosis  Delineate what a diagnosis implies clinically and societally  Define the logic by which diagnoses are made  Determine what affects how reliable our diagnoses can be  Define the role of the laboratory test in the diagnostic process
SOLUTION PART II Application of Evidence Based Medicine [EBM] to the implementation and management of laboratory methods and instrumentation Is the best approach to achieving an: Acceptable Balance Between benefits and risks when applying complex diagnostic modalities to our patient’s best interests. That is: Acceptable Risk
IMPLEMENTATION TO IMPLIMENT THIS WE WILL HAVE TO BE THE LEADERS Successful implementation of EBM requires both an educational and organizational effort to bring together the following activities:  CRITICAL REVIEW OF THE SCIENTIFIC LITERATURE  OBJECTIVE VALIDATION THROUGH LOCAL IMPLEMENTED ACTIVITIES  REALISTIC MEASUREMENT OF USEFULNESS OF THESE ACTIVITIES  MONITORING AND REPORTING OF TEST RELIABILITY AND ITS IMPACT Into a single integrated structure that assures close cooperation between those engaged in improving health care today:  Basic Researchers  Vendors  Clinicians  Laboratory  Government Regulatory Agencies
THE AXIS OF EVIDENCE BASED MEDICINE HOW DO WE BEGIN THE PROCESS? AXIS GOAL MEASURABLE OBJECTIVE SCIENCE ESTABLISH SCIENTIFIC FACT REPRODUCIBLE PROTOCOLS ANECDOTAL EVIDENCE VALIDATE AT THE PATIENT LEVEL MONITOR PATIENT OUTCOMES SOCIETAL REFERENCE FRAME FIT TO SOCIETAL EXPECTATIONS ACCEPTANCE BY PATIENTS VALUES GENERAL PATIENT The Definition of Evidence Based Medicine varies greatly. This is the most applicable
DEFINITIONS – DIAGNOSIS THEN A label applied to a cluster of seemingly related signs and symptoms that lead to the same or similar clinical outcomes. The components of a diagnosis were:  Presenting signs and symptoms  Natural progression of illness  Final outcome of recovery, chronic morbidity, and/or mortality Therefore, there was:  Little or no need for laboratory testing and  Little in the way of therapy and so  Little need to monitor patient response or to guide patient management.
DEFINITIONS – DIAGNOSIS NOW TODAY THE LABORATORY IS CENTRAL TO THE DIAGNOSTIC PROCESS A cluster of one or more of related illness complexes:  Signs  Symptom  Vital Measurements  Radiologic Studies  Pathologic diagnoses, and  Laboratory findings That leads to [to name a few]:  Clinical categorization  Prognosis  Therapy  Therapeutic monitoring for outcomes There is now great need for laboratory testing at every step of the process.
PURPOSE OF A TEST IS NOW MULTIFACETED FROM THIS WE CAN SEE THAT PRESENT METHODS OF VALIDATION ARE INADEQUATE There are eight main purposes of a laboratory test : PURPOSE DESCRIPTION SCREENING To identify a high risk subpopulation of people for definitive diagnostic workup RISK Stratification of subpopulation with increased probability of progression to disease DIAGNOSIS Establishing or ruling out a specific actionable cause for an illness complex CLASSIFICATION Determining the type of pathophysiologic mechanisms causing the illness PROGNOSIS Determining probable outcome if disease is left untreated STAGING Stratification for extent and aggressiveness of disease at time of diagnosis TREATMENT Determining most effective therapeutic modality MONITORING Observing outcomes to confirm diagnosis, document progress, and validate effectiveness of therapy Each type of activity will require a different level of minimum resolving power and reliability both acutely and over long periods of time between diagnosis and final disposition of the patient.
WHAT TYPE OF TEST IS NEEDED? IT IS CLEAR THAT TESTS MUST MATCH THEIR INTENDED PURPOSE(S) ACTIVITY WHAT HOW DETERMINED SCREENING Identify high risk population Highly sensitive tests DIAGNOSIS Filter out those without disease Highly specific tests PROGNOSIS Rank by risk for morbidity and mortality Highly discriminatory tests STAGING Rank by degree of appropriate therapy Tests that allow stratification MONITORING Determine therapeutic effect/diseases recurrence Tests that detect small changes Below is a brief example of what type of test would be needed for some of the purposes it might be applied to:
TEST VALIDITY There are four main types of laboratory test validation to consider TYPE DESCRIPTION ANALYTICAL Defines the ability of a test to reliably measure the analyte of interest CLINICAL Defines the ability of the test to establish if a disease state is present or not present UTILITY Defines the ability of a test to improve clinical outcomes IMPLICATIONS Societal implications and perceptions of the test’s value In essence, the validity of any particular laboratory test is determined by a number of interacting, synergistic, and conflicting issues that we face in today’s healthcare environment. So, the laboratory needs to understand all of these and to choose an optimal course of action when validating and monitoring test methodologies.
VALUE OF A TEST Once validity has been established the actual value of a test needs to be addressed. There are three main aspects to determining the value of a test:  Degree of association of a test result with a disease state  The effectiveness of a test in decision making activities  Utility in establishing/monitoring a the pathophysiologic state
There is very important clinical risk information implied by each test result that is acted upon by our clinicians often without fully understanding the implications…and circularity: BY DEFINITION, LAB TESTS THEMSELVES MEASURE CLINICAL RISK! SO; TESTS ARE A SOURCE OF RISK AND MEASURE RISK… TYPE RISK DESCRIPTION Reference ranges [intervals] LOW Population based screening for presence or risk of one or more diseases or physiologic states often in the asymptomatic patient. Medical Decision Points [MDP’s] MODERATE Statistical cut off for presence of a single disease or clinical state requiring timely treatment or modification of treatment - INSULIN Alert [Critical/Panic] Values HIGH Physiologically dangerous levels of an analyte requiring immediate evaluation and possibly intervention – OVER INSULINIZATION. How a test is applied for each level of risk is based on the following:  Clinical trials in the literature and/or personal anecdotal experience  Practice standards boards and deemed institutions  Cost benefit analysis by insurers and risk managers This determines the degree of reliability a test must have so that clinicians have confidence it meets their standard of Acceptable Risk
CATEGORIZATION OF RISK IN LABORATORY TESTING IS CRITICAL Risk specifically related to a laboratory testing is determined at three levels: IN THIS PRESENTATION WE WILL SEE HOW THIS CAN AFFECT US THEORETICAL OPERATIONAL TECHNICAL Analyte Significance Sensitivity Accuracy Relative Value Specificity Precision Basic Research Clinical Trials Laboratory Use AVOID THE WRONG TEST PREVENT INAPPROPRIATE USE MITIGATE EFFECT OF ANALYTICAL ERROR NOTE: ALTHOUGH THERE IS OVERLAP BETWEEN EACH LEVEL The type of risk that should be managed:  AVOIDANCE Is this test even worth commercializing?  PREVENTION If so what do we need to do to minimize adverse events?*  MITIGATION Where prevention is not enough can we monitor and correct? IS THIS WHAT ACTUALLY HAPPENS? *While maximizing utility value in diagnosis and management
ASSESSMENT OF CLINICAL VALUE FOR SYMMETRICAL BINARY MEDICAL DECISION POINT 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 Frequency Distribution of Dichotomous Patient Population A1c No Diabetes Type II Diabetes TRUE NEGATIVES NOTE: In a perfect world the two populations are symmetrical and equal in prevalence with very small overlap leading to high sensitivity and specificity as well as positive and negative predictive value TRUE POSITIVES FALSE POSITIVES FALSE NEGATIVES IN ACTUAL FACT, THE TWO DISTRIBUTIONS ARE NOTHING LIKE THIS
ASSESSMENT OF CLINICAL VALUE FOR ASYMETRICAL BINARY MEDICAL DECISION POINT 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 Frequency Distribution of Dichotomous Patient Population A1c No Diabetes Type II Diabetes NOTE: Although small on this graph, the absolute number of false positives will be unacceptably high when the MDP is set at the intersection of the two distribution curves. TRUE NEGATIVES TRUE POSITIVES FALSE POSITIVES FALSE NEGATIVES SO WHERE SHOULD THE MDP BE PLACED?
THE EFFECT OF TEST BIAS AND IMPRECISION THEREFORE IT IS CRITICAL THAT THE LABORTORY ESTIMATE RELIABILITY AND REPORT THIS Assuming that we have an optimally placed MDP: Systematic upward bias of A1c will: Increase false positive diagnosis and therefore:  Systematic increase in diagnosis of healthy persons leading to  Increase glycemic therapy leading to  Increased hypoglycemic events and short term morbidity and mortality Systematic downward bias of A1c will: Increase false negative diagnoses and therefore:  Systematic decrease in diagnosis of diabetic persons leading to  Decrease glycemic therapy leading to  Increased hyperglycemic events and long term morbidity and mortality
THE RESPONSIBILITY OF OUR CLINICIANS THEREFORE IT IS CRITICAL THAT THE LABORTORY EDUCATE OUR CLINICIANS OF THIS NEED We can estimate and report out reliability data until the cows come home. However, until our clinicians:  Recognize the significance of this data and  Begin to critically evaluate the establishment of Medical Decision Points We will not have the critical information regarding the actual risk for adverse outcomes that will occur for any combination of:  Placement of the Medical Decision Point  Resulting Sensitivity and Specificity  Prevalence of the Disease being Diagnosed  Types of Adverse Outcomes at Risk  Frequency of Adverse Outcomes at each MDP/Reliability pair  Severity of each Adverse Outcome So we will not be able to reliably establish Acceptable Risk
AND ASSIGNMENT OF THE MDP ESTABLISHES CLINIAL RISK(S) TAKEN WE NEED TO KNOW WHAT THE CLINICAL RISK IS BEYOND THE MDP  Clinical trials determine that, an analyte rises the number of adverse outcomes rises.  However, the trials do not determine specifically which patient is at risk.  At any level there will be a certain incidence of events.  As Upper MDP is moved  intervention will be increased   outcome1/  Outcome2  As Upper MDP is moved  intervention will be decreased   outcome1/ Outcome2 Risk for a Outcome 1 Risk for Outcome 2 POPULATION FREQUENCY ANALYTE Risk for a Outcome1 Risk for Outcome2 Medical Target Point Medical Decision Point to Increase Intervention Zone of Balanced Risk Medical Decision Point to Decrease Intervention An idealized presentation with a Gaussian distribution of patient test results for any particular intervention assuming all risks are of equal severity.
THE MDP DOES NOT DEFINE AN ALL OR NOTHING CLINICAL RISK WHAT IF THE DISTRIBUTION OF RISK ISN’T GAUSSIAN? Not all patients will have an event the moment their analyte rises above the upper MDP. Instead, there will be a distribution of significant events occurring over a range above the MDP with a maximum incidence of events beyond which there are less and less patients who have not yet had a bleed. Distribution of Risk for a Outcome1 OUTCOME 1 FREQUENCY ANALYTE Medical Target Point Medical Decision Point to Decrease Intervention so as to Maximize Benefit while Minimizing Risk of Outcome1 An idealized presentation with the affect of comorbid states not considered.
THE DISTRIBUTION OF ADVERSE EVENTS DEFINES CLINICAL RISK COMPARE THIS TO SKEWING IN THE OPPOSITE DIRECTION If the distribution of adverse events is this way then Acceptable Total Error must be very small. Otherwise even a small upward bias will greatly increase the risk for Outcome 1. Distribution of Risk for Outcome 1 OUTCOME 1 FREQUENCY ANALYTE Medical Target Point Medical Decision Point to Decrease Intervention so as to Maximize Benefit while Minimizing Risk of Outcome 1 An idealized presentation with the affect of comorbid states not considered. .
THE DISTRIBUTION OF ADVERSE EVENTS DEFINES CLINICAL RISK AND WE NEED TO KNOW WHAT THE CUMMULATIVE RISK IS! If the distribution of adverse events is skewed this way then the Acceptable Total Error can be much larger because Even a moderate upward bias will not greatly increase the risk for a bleed. Distribution of Risk for Outcome 1 OUTCOME 1 FREQUENCY ANALYTE Medical Target Point Medical Decision Point to Decrease Intervention so as to Maximize Benefit while Minimizing Risk of Outcome 1 An idealized presentation with the affect of comorbid states not considered.
CUMMULATIVE RISK PROVIDES A MEANS OF CLINICAL DECISION MAKING FINALLY, A USEFUL VISUAL CONCEPTUALIZATION OF RISK Note, depending on the width and height of the “Outcome” curve, there will be a point of inflection and asymptotic rise. While attempting to maximize benefits of therapy, the MDP should to be placed before this rise to prevent the majority of adverse outcomes. Finally, we have a means of determining more exactly a maximal total error we can tolerate OUTCOME 1 FREQUENCY ANALYTE 1.0 2.5 3.5 Medical Decision Point to Decrease Intervention 2.0 An idealized presentation with a Gaussian distribution of bleeding risk with distribution of ANALYTE Medical Target Point Cumulative Clinically Observed Frequency for Outcome 1 In Addition, the steepness of the slope of the Cumulative Distribution will Greatly Affect Degree of Risk at the MDP. Distribution of Laboratory Results Around the MDP can be compared to risk frequency to determine acceptability.
A CLOSE UP OF THE EFFECT OF IMPRECISION TAKEN ALONE THIS SHOWS TWO RISK COMPONENTS – CLINICAL & LABORATORY Below is the case where there is no bias but significant imprecision causing many results to lie far from the actual patient value that, in this case, lies just above the MDP. If we know the actual frequency at any point then we can determine overall risk. Cumulative Clinically Observed Frequency for Outcome 1 OUTCOME 1 FREQUENCY ANALYTE An idealized presentation with a Gaussian distribution of test results. Distribution of Laboratory Results Around the Patient Value 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Actual Patient ANALYTE Medical Decision Point to Decrease Intervention Cumulative Frequency of Patients with Test Results that are Erroneously Below the MDP Placing them at Risk for Over Treatment and Outcome 1. Cumulative Frequency of Patients with Test Results that are Erroneously High and May Place them at Risk for Under Treatment and Outcome 2. Frequency for Outcome 1 at 1 and 2 SD’s Away from the MDP can be Calculated .
A CLOSE UP OF THE EFFECT OF BIAS TAKEN ALONE THIS SHOWS TWO RISK COMPONENTS – CLINICAL & LABORATORY Below is the case where there is significant bias but no significant imprecision causing many results to lie to one side of actual patient value that, in this case, above the MDP. This type of error leads to systematic over treatment and, therefore  risk for a Outcome 1. Cumulative Clinically Observed Frequency for Outcome 1 OUTCOME 1 FREQUENCY ANALYTE An idealized presentation with a Gaussian distribution of test results. Distribution of Laboratory Results Around the Patient Value 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Actual Patient ANALTYE Medical Decision Point to Decrease Intervention Cumulative Frequency of Patients with Test Results that are Systematically Below the MDP Placing them at Risk for Over Treatment and Outcome 1. Frequency for a Outcome 1 at 1 and 2 SD’s Away from the MDP can be Calculated .
ASSESSMENT OF RISKS – CUMULATIVE FREQUENCY OF EVENTS/OUTCOMES: A1C 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 .20 .30 .40 .50 .60 .70 .80 .90 Cumulative Frequency of Event/Outcome A1c 1.00 .10 Hypoglycemic Events Hyperglycemic Events Overlap Zone NOTE: Access to this quantitative data would allow for us to determine an overlap zone for placement of our Medical Decision Point to minimize one or the other Event/Outcome as needed. CUMULATIVE FREQUENCY IS CRITICAL FOR RATIONAL DECISION MAKING
ASSESSMENT OF RISKS – CUMULATIVE DEGREE OF SEVERITY: A1C 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 .20 .30 .40 .50 .60 .70 .80 .90 Cumulative Degree of Severity [Severity x Frequency] A1c 1.00 .10 Severity of Hypoglycemic Event Severity of Hyperglycemic Outcome(s) Overlap Zone NOTE: Developing a useful ranking system would allow for us to determine an overlap zone for placement of our Medical Decision Point to minimize severity. Mortality Morbidity CUMULATIVE SEVERITY IS CRITICAL FOR PERSONAL AND SOCIETAL DECISION MAKING
In order to fully integrate the clinical and laboratory rolls in choosing and utilizing test methodologies in the diagnosis and treatment of disease we need to revisit that part of FMEA that causes us the greatest problem: Calculating a Risk Priority Number [RPN] to assign levels of severity upon which decision making could be more realistically be based in defining Acceptable Risk. Although there is no one best way to approach this matter, we can create models upon which to base further clinical and laboratory research to validate their applicability. Below is an approach I recommend. I don’t claim that it is necessarily the best. However, if nothing else, this approach clearly delineates all of the problems we face in any particular medical activity whether Point-of-Care Testing or in utilizing other studies in classifying a patient’s clinical status upon which to base further diagnostic and/or therapeutic efforts. By doing this we define the limits of our knowledge and our capabilities CLINICAL BASIS FOR DETERMINING SEVERITY OF A RISK CONSIDER A COMBINED QUALITATIVE/QUANTITATIVE MODEL
We really need a way to easily categorize risk by clearly applicable criteria that can incorporate clinician perception. I find that the following two descriptive sets, when combined as a matrix, provide a more operationally useful approach: TYPE OF RISK: [Including but not necessarily limited to]  Pain, Discomfort, Malaise, GI Upset, Etc.  Physical Damage – Structural and Cosmetic  Functional Limitation – Orthopedic, Muscular, Neurologic, Cognitive, Immunologic, etc.  Physiologic – Endurance, Handling of Catabolic Waste, Organ Functionality  Psychologic – The Impact of all of the Above Factors DEGREE OF ADVERSITY FOR EACH TYPE:  Morbidity – Temporary – Relatively Short Lived with Full Resolution  Morbidity – Persistent – Intermediate Term Course with Full or Partial Resolution  Morbidity – Permanent – Long Term Course with Partial or No Resolution  Mortality – Distant, Delayed, or Immediate – Due to Disease or Complications COMPONENTS OF A SEVERITY OF A RISK ASSESSMENT THESE RISK AXIS’ PROVIDE A MEANS OF OBTAINING CLINICIAN RANKING
Below is my proposed stratification of severity creating a 10 point scale that is open ended allowing the medical community and society to define the descriptive terms: PROPOSED MODEL FOR DETERMINING SEVERITY OF A RISK RANKING SYSTEMS OFTEN IMPLY A LINEAR PROGRESSION. IS THIS APPROPRIATE? RANK RISK STRATIFICATION DESCRIPTION 0 No significant affect Morbidity - Temporary Pain, Physical, Functional, Physiologic, Psychologic 1 Mild 2 Moderate 3 Marked Morbidity – Persistent Pain, Physical, Functional, Physiologic, Psychologic 4 Mild 5 Moderate 6 Marked Morbidity – Permanent Pain, Physical, Functional, Physiologic, Psychologic 7 Mild 8 Moderate 9 Marked Mortality By Disease or By Complications of Therapy 10 Distant 10 Delayed 10 Immediate
PROPOSED TRANSLATION OF THE SEVERITY “CURVE” IT SHOULD LOOK LIKE FALLING OFF A CLIFF OR OVER A WATER FALL The severity value assigned to each succeeding severity rank should increase at an accelerating rate until the value reaches its maximum at death. If we also translate frequency from linear to a power function then multiplying them with detection rate to obtain a Risk Priority Number would result in a more realistic valuation to define Acceptable Risk. By applying a reversed power function such as a parabolic or other similar function we more accurately assign a severity value to each rank. From this an FMEA Risk Priority Number can be calculated to use in positioning the Medical Decision Point and, along with the cumulative distribution of adverse outcomes, develop a Maximal Total Error Target. SEVERITY VALUE SEVERITY RANK A Linear valuation would not match the perceived loss on the part of patients, family, and society. It over values mild levels of severity and under values marked levels of severity. An idealized presentation with the affect of comorbid states not considered. 0 1 2 3 4 5 6 7 8 9 10 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Below is a proposed set of steps our clinicians should take in establishing Acceptable Risk before we can attempt to determine what level of Acceptable Analytical Error we need to meet in choosing and implementing a test methodology: CLINICAL BASIS FOR DETERMINING ACCEPTABLE RISK ONCE THIS IS ESTABLISHED THEN CRITERIA FOR TEST METHODOLOGY IS IN PLACE # TASK CLINICAL RESPONSIBILITIES 1 Define disease to be investigated clinically. 2 Define diagnostic/therapeutic criteria. 3 Determine optimal laboratory analyte to test criteria clinically. 4 Define therapeutic modalities that will be initiated or withheld based on test results. 5 Determine benefits of therapy versus withholding therapy. 6 Determine risks of therapy versus withholding therapy. 7 Assign Severity Rank and determine Severity Value. 8 Measure frequency of adverse outcomes and determine the Risk Priority Number. 9 Determine affects of comorbid states on frequency and on severity as needed. 10 Assign Medical Decision Point(s) based on Cost/Benefit analysis of above information. 11 Determine maximum variance around the MDP that avoids Unacceptable Risk. 12 Establish maximum total Acceptable Analytical Error for the laboratory. LABORATORY RESPONSIBILITIES 13 Evaluate Available Testing Systems for Acceptable Analytical Methodology. 14 Carry out appropriate validation studies and reject or implement test methodology.
RECOMMENDATIONS FOR THE LABORATORY REGARDLESS OF HOW THE CLINICAL ASPECTS OF ACCEPTABLE RISK ARE HANDLED
VALIDATION OF INSTRUMENTS SHOULD INCLUDE: CLINICAL:  Determine specifically what your facility is going to use the test for.  Determine the environment in which the instrument will be used.  Establish clinical standards for Acceptable Risk for that facility if possible. LABORATORY:  Validate by linear regression/bias plot to establish reliability over a range of values  Validate by modified Cohen’s statistic at MDP’s to estimate our part of risk COOPERATIVE:  Determine if the instrument meets clinician perception for Acceptable Risk  If so, implement with full orientation, training, and procedural documentation  Initiate period of full monitoring to assure operational reliability  Monitor estimated technical error to identify any significant shifts  Validate clinical protocol to assure it works in THAT facility. RECOMMENDED STEPS IN ESTABLISHING POC TESTING AN INTEGRATED APPROACH BETWEEN CLINICAL AND LAB STAFF IS BEST
Ideally, the laboratory should travel down the following path to the point where it can evaluate clinical risk with great confidence. RECOMMENDED STEPS IN VALIDATING TESTING METHODOLOGIES THIS REQUIRES AN INTEGRATED SOFTWARE APPLICATION TO ASSIST US IN VALIDATION STEP QUESTION TO ANSWER TYPE STATISTICAL TOOL RECOMMENDED* 1 Is the data unimodal? MDP Data Distribution Plot 2 Is the data normally distributed? MDP χ2-Test 3 Is estimated precision low? MDP SD CV / Range 4 Is estimated accuracy high? MDP Reference – Mean / Reference - Median 5 Are the SD’s the same? COM F-Test 6 Are the Means the same? COM T-Test 7 Is there a linear relationship? COM Regression 8 What is the range of linearity? COM Bias Plot evaluation 9 What is the overall reliability? COM Correlation Coefficient 10 What is the concordance at MDP’s? MDP Simplified Cohen’s Kappa * NOTE: There are many others in the literature.
SUGGESTED FLOW CHART TO ASSURE CLINICAL RELIABILITY ACCEPTABLE RISK SHOULD DRIVE CLINICAL AND LAB TEST MONITORING Selection of Analyte Selection of Test Methodology Clinical Application Defined Risks for Each State Defined Clinical States Defined Acceptable Risk Defined Technical Error Limits Defined Validation of Methodology Acceptable Performance? YES NO Develop Utilization Policies Institute Adjustments if Needed Write Technical Procedures Publish Estimated Reliability Orient and Train Users Monitor Analytic Reliability Monitor Clinical Results Carry out Ongoing Upgrades Retire at End of Life Cycle Risk Distribution(s) Defined
PUTTING IT ALL TOGETHER AN INTEGRATED APPROACH IS BEST In his 1989 article on Controlling Error in Laboratory Testing Donald Forman, Ph.D puts it all together in an integrated overview of problems which is applicable to all Clinical Laboratory Testing: “…correct interpretation of test results requires knowledge of analytical and biological as well as pathophysiological sources of variation, their expected magnitude, and the time course over which changes can occur in health and disease.” In just one sentence Dr. Forman includes all the critical factors we as laboratorians and our clinicains must consider when implementing, measuring, reporting, and applying any test analyte result. The operational and technical problems we face in doing so are very significant. Add in cultural resistance on the part of laboratorians and physicians and you have a serious obstacle to improvement in the laboratory. Forman, D.T., MLO Supplement 1989.
THERE IS MUCH STILL TO DO: It is clear that, today despite miraculous advances in the knowledge and application of that knowledge to the clinical laboratory we are still faced with significant challenges when it comes to the appropriate validation of the tests we place in the hands of our clinical staff. KNOWLEDGE AND UNDERSTANDING IS LACKING: Presently our clinicians are not educating themselves adequately in the fundamental science underlying these tools in order to understand in detail their strengths and weaknesses and this has lead to health care by cook book and not medicine by clinical acumen placing the laboratory in a difficult situation for expected accuracy and precision. WHAT IS TO BE DONE? If we are going to maintain our position as a highly regarded profession that is dedicated to service to our clinicians and patients then we must begin to rethink how we apply all this technology to our efforts at clinical evaluation and management. Otherwise, soon, we will be submerged under a collective system of government regulations and accreditations standards that may do more harm than good. CONCLUSIONS AND WARNINGS FINIS! QUESTIONS?
REFERENCES REFER TO THE SEPARATE REFERENCE LISTING

More Related Content

[Typ]Presentation[Sbj]LaboratoryDiagnosisDefined[Dte]20131028

  • 1. THE LABORATORY’S ROLE IN DIAGNOSIS AND DIAGNOSTIC DECISION MAKING Mark Gusack, M.D., Staff Pathologist VA Medical Center, Huntington, WV Adjunct Clinical Professor Marshall University Medical School NORTHEAST LABORATORY CONFERENCE 15 OCT 2013 RISK
  • 2. Mark Gusack, M.D., OCD [Overly Concerned Doctor] Staff Pathologist, Pathology and Laboratory Medicine Service VA Medical Center Huntington, WV 25704 mark.gusack2@va.gov SPEAKER Dr. Gusack has almost 40 years experience in the Laboratory field starting as a Nuclear Medicine Technologist in the early 1970’s, then working as a clinical engineer, and then becoming a physician and pathologist. He is AP/CP boarded, has held positions in a variety of hospital and reference based laboratories as a medical director and as staff pathologist. During this time he has also been a consultant and practiced as a Licensed Health Care Risk Manager in Florida. Dr. Gusack has been involved with all aspects of laboratory development and management including startup, licensing, as well as designing integrated management systems for clinical laboratories. The opinions expressed in this presentation are those of the author and do not necessarily represent those of the Department of Veterans Affairs
  • 3. OUR MISSION HOW DO WE CARRY OUT THIS MISSION? OUR MISION IS TO PROVIDE THE VERY BEST CARE TO OUR PATIENTS
  • 4. SITUATION – PART I HOW HAS THIS AFFECTED WHAT WE DO IN THE LABORATORY? The meaning of diagnosis has changed radically over the past 60 – 70 years. THEN:  In the past, there were few diagnostic tests and fewer effective treatments.  A diagnosis was a label placed on a set of signs and symptoms with a prognosis.  Since most therapy was ineffective error in diagnosis was rarely a critical issue. NOW:  Since that time our fund of knowledge has grown prodigiously.  Our ability to apply this knowledge to health care has transformed medicine.  An error in diagnosis has serious consequences given more effective therapy. THE CONSEQUENCES:  The patient can be harmed or die due to a false positive or negative diagnosis.  Increased morbidity and mortality in a population can cost a society greatly.  Increased numbers of failure in the face of unprecedented success.
  • 5. SITUATION – PART II SO WE HAVE INITIATED A CYCLE OF:  Increased subdivision of older diagnoses into numerous newer diseases.  Increase capacity to make an accurate diagnoses.  Increased ability to effectively treat more diseases. LEADING TO:  A dramatic increase in the complexity of diagnostic medicine.  A dramatic increase in the perceived impact of a diagnosis. CONSEQUENCES:  Increased expectations for a “perfect” outcome.  Increased apparent error in diagnosis.  Increased perception of harm by the patient and society. Now standards of Acceptable Risk are much more stringent
  • 6. SITUATION – PART III – THE RISE OF CHRONIC DISEASE THIS HAS DRAMATICALLY CHANGED THE MEDICAL PROFESSION Today most acute diseases have been eliminated through: Civil Engineering: Waste, water, and food safety management Improved Diet: Improving our capacity to survive an illness Modern Medicine: Improved therapies to cure acute illness As a result today we are dealing primarily with:  Chronic, progressive, debilitating diseases for which  There is often no definitive curative therapy which  Cost far more to treat than acute illnesses which  Escalate over time as the patient progressively declines in health Present national budget problems are due,to a great degree to this progression And what WE do will determine, to a great degree the final outcome.
  • 7. PROBLEM How can we reduce the impact of Systematic Complexity of the Diagnostic Process on how we generate test results in order to assure establishing: Acceptable Risk For our Patients?
  • 8. SOLUTION PART I THIS IS NOT GOING TO BE EASY In order to establish Acceptable Risk we need to:  Define what the term diagnosis means  Understand the difference between the concept of a diagnosis and the process of making a diagnosis  Delineate what a diagnosis implies clinically and societally  Define the logic by which diagnoses are made  Determine what affects how reliable our diagnoses can be  Define the role of the laboratory test in the diagnostic process
  • 9. SOLUTION PART II Application of Evidence Based Medicine [EBM] to the implementation and management of laboratory methods and instrumentation Is the best approach to achieving an: Acceptable Balance Between benefits and risks when applying complex diagnostic modalities to our patient’s best interests. That is: Acceptable Risk
  • 10. IMPLEMENTATION TO IMPLIMENT THIS WE WILL HAVE TO BE THE LEADERS Successful implementation of EBM requires both an educational and organizational effort to bring together the following activities:  CRITICAL REVIEW OF THE SCIENTIFIC LITERATURE  OBJECTIVE VALIDATION THROUGH LOCAL IMPLEMENTED ACTIVITIES  REALISTIC MEASUREMENT OF USEFULNESS OF THESE ACTIVITIES  MONITORING AND REPORTING OF TEST RELIABILITY AND ITS IMPACT Into a single integrated structure that assures close cooperation between those engaged in improving health care today:  Basic Researchers  Vendors  Clinicians  Laboratory  Government Regulatory Agencies
  • 11. THE AXIS OF EVIDENCE BASED MEDICINE HOW DO WE BEGIN THE PROCESS? AXIS GOAL MEASURABLE OBJECTIVE SCIENCE ESTABLISH SCIENTIFIC FACT REPRODUCIBLE PROTOCOLS ANECDOTAL EVIDENCE VALIDATE AT THE PATIENT LEVEL MONITOR PATIENT OUTCOMES SOCIETAL REFERENCE FRAME FIT TO SOCIETAL EXPECTATIONS ACCEPTANCE BY PATIENTS VALUES GENERAL PATIENT The Definition of Evidence Based Medicine varies greatly. This is the most applicable
  • 12. DEFINITIONS – DIAGNOSIS THEN A label applied to a cluster of seemingly related signs and symptoms that lead to the same or similar clinical outcomes. The components of a diagnosis were:  Presenting signs and symptoms  Natural progression of illness  Final outcome of recovery, chronic morbidity, and/or mortality Therefore, there was:  Little or no need for laboratory testing and  Little in the way of therapy and so  Little need to monitor patient response or to guide patient management.
  • 13. DEFINITIONS – DIAGNOSIS NOW TODAY THE LABORATORY IS CENTRAL TO THE DIAGNOSTIC PROCESS A cluster of one or more of related illness complexes:  Signs  Symptom  Vital Measurements  Radiologic Studies  Pathologic diagnoses, and  Laboratory findings That leads to [to name a few]:  Clinical categorization  Prognosis  Therapy  Therapeutic monitoring for outcomes There is now great need for laboratory testing at every step of the process.
  • 14. PURPOSE OF A TEST IS NOW MULTIFACETED FROM THIS WE CAN SEE THAT PRESENT METHODS OF VALIDATION ARE INADEQUATE There are eight main purposes of a laboratory test : PURPOSE DESCRIPTION SCREENING To identify a high risk subpopulation of people for definitive diagnostic workup RISK Stratification of subpopulation with increased probability of progression to disease DIAGNOSIS Establishing or ruling out a specific actionable cause for an illness complex CLASSIFICATION Determining the type of pathophysiologic mechanisms causing the illness PROGNOSIS Determining probable outcome if disease is left untreated STAGING Stratification for extent and aggressiveness of disease at time of diagnosis TREATMENT Determining most effective therapeutic modality MONITORING Observing outcomes to confirm diagnosis, document progress, and validate effectiveness of therapy Each type of activity will require a different level of minimum resolving power and reliability both acutely and over long periods of time between diagnosis and final disposition of the patient.
  • 15. WHAT TYPE OF TEST IS NEEDED? IT IS CLEAR THAT TESTS MUST MATCH THEIR INTENDED PURPOSE(S) ACTIVITY WHAT HOW DETERMINED SCREENING Identify high risk population Highly sensitive tests DIAGNOSIS Filter out those without disease Highly specific tests PROGNOSIS Rank by risk for morbidity and mortality Highly discriminatory tests STAGING Rank by degree of appropriate therapy Tests that allow stratification MONITORING Determine therapeutic effect/diseases recurrence Tests that detect small changes Below is a brief example of what type of test would be needed for some of the purposes it might be applied to:
  • 16. TEST VALIDITY There are four main types of laboratory test validation to consider TYPE DESCRIPTION ANALYTICAL Defines the ability of a test to reliably measure the analyte of interest CLINICAL Defines the ability of the test to establish if a disease state is present or not present UTILITY Defines the ability of a test to improve clinical outcomes IMPLICATIONS Societal implications and perceptions of the test’s value In essence, the validity of any particular laboratory test is determined by a number of interacting, synergistic, and conflicting issues that we face in today’s healthcare environment. So, the laboratory needs to understand all of these and to choose an optimal course of action when validating and monitoring test methodologies.
  • 17. VALUE OF A TEST Once validity has been established the actual value of a test needs to be addressed. There are three main aspects to determining the value of a test:  Degree of association of a test result with a disease state  The effectiveness of a test in decision making activities  Utility in establishing/monitoring a the pathophysiologic state
  • 18. There is very important clinical risk information implied by each test result that is acted upon by our clinicians often without fully understanding the implications…and circularity: BY DEFINITION, LAB TESTS THEMSELVES MEASURE CLINICAL RISK! SO; TESTS ARE A SOURCE OF RISK AND MEASURE RISK… TYPE RISK DESCRIPTION Reference ranges [intervals] LOW Population based screening for presence or risk of one or more diseases or physiologic states often in the asymptomatic patient. Medical Decision Points [MDP’s] MODERATE Statistical cut off for presence of a single disease or clinical state requiring timely treatment or modification of treatment - INSULIN Alert [Critical/Panic] Values HIGH Physiologically dangerous levels of an analyte requiring immediate evaluation and possibly intervention – OVER INSULINIZATION. How a test is applied for each level of risk is based on the following:  Clinical trials in the literature and/or personal anecdotal experience  Practice standards boards and deemed institutions  Cost benefit analysis by insurers and risk managers This determines the degree of reliability a test must have so that clinicians have confidence it meets their standard of Acceptable Risk
  • 19. CATEGORIZATION OF RISK IN LABORATORY TESTING IS CRITICAL Risk specifically related to a laboratory testing is determined at three levels: IN THIS PRESENTATION WE WILL SEE HOW THIS CAN AFFECT US THEORETICAL OPERATIONAL TECHNICAL Analyte Significance Sensitivity Accuracy Relative Value Specificity Precision Basic Research Clinical Trials Laboratory Use AVOID THE WRONG TEST PREVENT INAPPROPRIATE USE MITIGATE EFFECT OF ANALYTICAL ERROR NOTE: ALTHOUGH THERE IS OVERLAP BETWEEN EACH LEVEL The type of risk that should be managed:  AVOIDANCE Is this test even worth commercializing?  PREVENTION If so what do we need to do to minimize adverse events?*  MITIGATION Where prevention is not enough can we monitor and correct? IS THIS WHAT ACTUALLY HAPPENS? *While maximizing utility value in diagnosis and management
  • 20. ASSESSMENT OF CLINICAL VALUE FOR SYMMETRICAL BINARY MEDICAL DECISION POINT 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 Frequency Distribution of Dichotomous Patient Population A1c No Diabetes Type II Diabetes TRUE NEGATIVES NOTE: In a perfect world the two populations are symmetrical and equal in prevalence with very small overlap leading to high sensitivity and specificity as well as positive and negative predictive value TRUE POSITIVES FALSE POSITIVES FALSE NEGATIVES IN ACTUAL FACT, THE TWO DISTRIBUTIONS ARE NOTHING LIKE THIS
  • 21. ASSESSMENT OF CLINICAL VALUE FOR ASYMETRICAL BINARY MEDICAL DECISION POINT 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 Frequency Distribution of Dichotomous Patient Population A1c No Diabetes Type II Diabetes NOTE: Although small on this graph, the absolute number of false positives will be unacceptably high when the MDP is set at the intersection of the two distribution curves. TRUE NEGATIVES TRUE POSITIVES FALSE POSITIVES FALSE NEGATIVES SO WHERE SHOULD THE MDP BE PLACED?
  • 22. THE EFFECT OF TEST BIAS AND IMPRECISION THEREFORE IT IS CRITICAL THAT THE LABORTORY ESTIMATE RELIABILITY AND REPORT THIS Assuming that we have an optimally placed MDP: Systematic upward bias of A1c will: Increase false positive diagnosis and therefore:  Systematic increase in diagnosis of healthy persons leading to  Increase glycemic therapy leading to  Increased hypoglycemic events and short term morbidity and mortality Systematic downward bias of A1c will: Increase false negative diagnoses and therefore:  Systematic decrease in diagnosis of diabetic persons leading to  Decrease glycemic therapy leading to  Increased hyperglycemic events and long term morbidity and mortality
  • 23. THE RESPONSIBILITY OF OUR CLINICIANS THEREFORE IT IS CRITICAL THAT THE LABORTORY EDUCATE OUR CLINICIANS OF THIS NEED We can estimate and report out reliability data until the cows come home. However, until our clinicians:  Recognize the significance of this data and  Begin to critically evaluate the establishment of Medical Decision Points We will not have the critical information regarding the actual risk for adverse outcomes that will occur for any combination of:  Placement of the Medical Decision Point  Resulting Sensitivity and Specificity  Prevalence of the Disease being Diagnosed  Types of Adverse Outcomes at Risk  Frequency of Adverse Outcomes at each MDP/Reliability pair  Severity of each Adverse Outcome So we will not be able to reliably establish Acceptable Risk
  • 24. AND ASSIGNMENT OF THE MDP ESTABLISHES CLINIAL RISK(S) TAKEN WE NEED TO KNOW WHAT THE CLINICAL RISK IS BEYOND THE MDP  Clinical trials determine that, an analyte rises the number of adverse outcomes rises.  However, the trials do not determine specifically which patient is at risk.  At any level there will be a certain incidence of events.  As Upper MDP is moved  intervention will be increased   outcome1/  Outcome2  As Upper MDP is moved  intervention will be decreased   outcome1/ Outcome2 Risk for a Outcome 1 Risk for Outcome 2 POPULATION FREQUENCY ANALYTE Risk for a Outcome1 Risk for Outcome2 Medical Target Point Medical Decision Point to Increase Intervention Zone of Balanced Risk Medical Decision Point to Decrease Intervention An idealized presentation with a Gaussian distribution of patient test results for any particular intervention assuming all risks are of equal severity.
  • 25. THE MDP DOES NOT DEFINE AN ALL OR NOTHING CLINICAL RISK WHAT IF THE DISTRIBUTION OF RISK ISN’T GAUSSIAN? Not all patients will have an event the moment their analyte rises above the upper MDP. Instead, there will be a distribution of significant events occurring over a range above the MDP with a maximum incidence of events beyond which there are less and less patients who have not yet had a bleed. Distribution of Risk for a Outcome1 OUTCOME 1 FREQUENCY ANALYTE Medical Target Point Medical Decision Point to Decrease Intervention so as to Maximize Benefit while Minimizing Risk of Outcome1 An idealized presentation with the affect of comorbid states not considered.
  • 26. THE DISTRIBUTION OF ADVERSE EVENTS DEFINES CLINICAL RISK COMPARE THIS TO SKEWING IN THE OPPOSITE DIRECTION If the distribution of adverse events is this way then Acceptable Total Error must be very small. Otherwise even a small upward bias will greatly increase the risk for Outcome 1. Distribution of Risk for Outcome 1 OUTCOME 1 FREQUENCY ANALYTE Medical Target Point Medical Decision Point to Decrease Intervention so as to Maximize Benefit while Minimizing Risk of Outcome 1 An idealized presentation with the affect of comorbid states not considered. .
  • 27. THE DISTRIBUTION OF ADVERSE EVENTS DEFINES CLINICAL RISK AND WE NEED TO KNOW WHAT THE CUMMULATIVE RISK IS! If the distribution of adverse events is skewed this way then the Acceptable Total Error can be much larger because Even a moderate upward bias will not greatly increase the risk for a bleed. Distribution of Risk for Outcome 1 OUTCOME 1 FREQUENCY ANALYTE Medical Target Point Medical Decision Point to Decrease Intervention so as to Maximize Benefit while Minimizing Risk of Outcome 1 An idealized presentation with the affect of comorbid states not considered.
  • 28. CUMMULATIVE RISK PROVIDES A MEANS OF CLINICAL DECISION MAKING FINALLY, A USEFUL VISUAL CONCEPTUALIZATION OF RISK Note, depending on the width and height of the “Outcome” curve, there will be a point of inflection and asymptotic rise. While attempting to maximize benefits of therapy, the MDP should to be placed before this rise to prevent the majority of adverse outcomes. Finally, we have a means of determining more exactly a maximal total error we can tolerate OUTCOME 1 FREQUENCY ANALYTE 1.0 2.5 3.5 Medical Decision Point to Decrease Intervention 2.0 An idealized presentation with a Gaussian distribution of bleeding risk with distribution of ANALYTE Medical Target Point Cumulative Clinically Observed Frequency for Outcome 1 In Addition, the steepness of the slope of the Cumulative Distribution will Greatly Affect Degree of Risk at the MDP. Distribution of Laboratory Results Around the MDP can be compared to risk frequency to determine acceptability.
  • 29. A CLOSE UP OF THE EFFECT OF IMPRECISION TAKEN ALONE THIS SHOWS TWO RISK COMPONENTS – CLINICAL & LABORATORY Below is the case where there is no bias but significant imprecision causing many results to lie far from the actual patient value that, in this case, lies just above the MDP. If we know the actual frequency at any point then we can determine overall risk. Cumulative Clinically Observed Frequency for Outcome 1 OUTCOME 1 FREQUENCY ANALYTE An idealized presentation with a Gaussian distribution of test results. Distribution of Laboratory Results Around the Patient Value 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Actual Patient ANALYTE Medical Decision Point to Decrease Intervention Cumulative Frequency of Patients with Test Results that are Erroneously Below the MDP Placing them at Risk for Over Treatment and Outcome 1. Cumulative Frequency of Patients with Test Results that are Erroneously High and May Place them at Risk for Under Treatment and Outcome 2. Frequency for Outcome 1 at 1 and 2 SD’s Away from the MDP can be Calculated .
  • 30. A CLOSE UP OF THE EFFECT OF BIAS TAKEN ALONE THIS SHOWS TWO RISK COMPONENTS – CLINICAL & LABORATORY Below is the case where there is significant bias but no significant imprecision causing many results to lie to one side of actual patient value that, in this case, above the MDP. This type of error leads to systematic over treatment and, therefore  risk for a Outcome 1. Cumulative Clinically Observed Frequency for Outcome 1 OUTCOME 1 FREQUENCY ANALYTE An idealized presentation with a Gaussian distribution of test results. Distribution of Laboratory Results Around the Patient Value 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Actual Patient ANALTYE Medical Decision Point to Decrease Intervention Cumulative Frequency of Patients with Test Results that are Systematically Below the MDP Placing them at Risk for Over Treatment and Outcome 1. Frequency for a Outcome 1 at 1 and 2 SD’s Away from the MDP can be Calculated .
  • 31. ASSESSMENT OF RISKS – CUMULATIVE FREQUENCY OF EVENTS/OUTCOMES: A1C 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 .20 .30 .40 .50 .60 .70 .80 .90 Cumulative Frequency of Event/Outcome A1c 1.00 .10 Hypoglycemic Events Hyperglycemic Events Overlap Zone NOTE: Access to this quantitative data would allow for us to determine an overlap zone for placement of our Medical Decision Point to minimize one or the other Event/Outcome as needed. CUMULATIVE FREQUENCY IS CRITICAL FOR RATIONAL DECISION MAKING
  • 32. ASSESSMENT OF RISKS – CUMULATIVE DEGREE OF SEVERITY: A1C 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 .20 .30 .40 .50 .60 .70 .80 .90 Cumulative Degree of Severity [Severity x Frequency] A1c 1.00 .10 Severity of Hypoglycemic Event Severity of Hyperglycemic Outcome(s) Overlap Zone NOTE: Developing a useful ranking system would allow for us to determine an overlap zone for placement of our Medical Decision Point to minimize severity. Mortality Morbidity CUMULATIVE SEVERITY IS CRITICAL FOR PERSONAL AND SOCIETAL DECISION MAKING
  • 33. In order to fully integrate the clinical and laboratory rolls in choosing and utilizing test methodologies in the diagnosis and treatment of disease we need to revisit that part of FMEA that causes us the greatest problem: Calculating a Risk Priority Number [RPN] to assign levels of severity upon which decision making could be more realistically be based in defining Acceptable Risk. Although there is no one best way to approach this matter, we can create models upon which to base further clinical and laboratory research to validate their applicability. Below is an approach I recommend. I don’t claim that it is necessarily the best. However, if nothing else, this approach clearly delineates all of the problems we face in any particular medical activity whether Point-of-Care Testing or in utilizing other studies in classifying a patient’s clinical status upon which to base further diagnostic and/or therapeutic efforts. By doing this we define the limits of our knowledge and our capabilities CLINICAL BASIS FOR DETERMINING SEVERITY OF A RISK CONSIDER A COMBINED QUALITATIVE/QUANTITATIVE MODEL
  • 34. We really need a way to easily categorize risk by clearly applicable criteria that can incorporate clinician perception. I find that the following two descriptive sets, when combined as a matrix, provide a more operationally useful approach: TYPE OF RISK: [Including but not necessarily limited to]  Pain, Discomfort, Malaise, GI Upset, Etc.  Physical Damage – Structural and Cosmetic  Functional Limitation – Orthopedic, Muscular, Neurologic, Cognitive, Immunologic, etc.  Physiologic – Endurance, Handling of Catabolic Waste, Organ Functionality  Psychologic – The Impact of all of the Above Factors DEGREE OF ADVERSITY FOR EACH TYPE:  Morbidity – Temporary – Relatively Short Lived with Full Resolution  Morbidity – Persistent – Intermediate Term Course with Full or Partial Resolution  Morbidity – Permanent – Long Term Course with Partial or No Resolution  Mortality – Distant, Delayed, or Immediate – Due to Disease or Complications COMPONENTS OF A SEVERITY OF A RISK ASSESSMENT THESE RISK AXIS’ PROVIDE A MEANS OF OBTAINING CLINICIAN RANKING
  • 35. Below is my proposed stratification of severity creating a 10 point scale that is open ended allowing the medical community and society to define the descriptive terms: PROPOSED MODEL FOR DETERMINING SEVERITY OF A RISK RANKING SYSTEMS OFTEN IMPLY A LINEAR PROGRESSION. IS THIS APPROPRIATE? RANK RISK STRATIFICATION DESCRIPTION 0 No significant affect Morbidity - Temporary Pain, Physical, Functional, Physiologic, Psychologic 1 Mild 2 Moderate 3 Marked Morbidity – Persistent Pain, Physical, Functional, Physiologic, Psychologic 4 Mild 5 Moderate 6 Marked Morbidity – Permanent Pain, Physical, Functional, Physiologic, Psychologic 7 Mild 8 Moderate 9 Marked Mortality By Disease or By Complications of Therapy 10 Distant 10 Delayed 10 Immediate
  • 36. PROPOSED TRANSLATION OF THE SEVERITY “CURVE” IT SHOULD LOOK LIKE FALLING OFF A CLIFF OR OVER A WATER FALL The severity value assigned to each succeeding severity rank should increase at an accelerating rate until the value reaches its maximum at death. If we also translate frequency from linear to a power function then multiplying them with detection rate to obtain a Risk Priority Number would result in a more realistic valuation to define Acceptable Risk. By applying a reversed power function such as a parabolic or other similar function we more accurately assign a severity value to each rank. From this an FMEA Risk Priority Number can be calculated to use in positioning the Medical Decision Point and, along with the cumulative distribution of adverse outcomes, develop a Maximal Total Error Target. SEVERITY VALUE SEVERITY RANK A Linear valuation would not match the perceived loss on the part of patients, family, and society. It over values mild levels of severity and under values marked levels of severity. An idealized presentation with the affect of comorbid states not considered. 0 1 2 3 4 5 6 7 8 9 10 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
  • 37. Below is a proposed set of steps our clinicians should take in establishing Acceptable Risk before we can attempt to determine what level of Acceptable Analytical Error we need to meet in choosing and implementing a test methodology: CLINICAL BASIS FOR DETERMINING ACCEPTABLE RISK ONCE THIS IS ESTABLISHED THEN CRITERIA FOR TEST METHODOLOGY IS IN PLACE # TASK CLINICAL RESPONSIBILITIES 1 Define disease to be investigated clinically. 2 Define diagnostic/therapeutic criteria. 3 Determine optimal laboratory analyte to test criteria clinically. 4 Define therapeutic modalities that will be initiated or withheld based on test results. 5 Determine benefits of therapy versus withholding therapy. 6 Determine risks of therapy versus withholding therapy. 7 Assign Severity Rank and determine Severity Value. 8 Measure frequency of adverse outcomes and determine the Risk Priority Number. 9 Determine affects of comorbid states on frequency and on severity as needed. 10 Assign Medical Decision Point(s) based on Cost/Benefit analysis of above information. 11 Determine maximum variance around the MDP that avoids Unacceptable Risk. 12 Establish maximum total Acceptable Analytical Error for the laboratory. LABORATORY RESPONSIBILITIES 13 Evaluate Available Testing Systems for Acceptable Analytical Methodology. 14 Carry out appropriate validation studies and reject or implement test methodology.
  • 38. RECOMMENDATIONS FOR THE LABORATORY REGARDLESS OF HOW THE CLINICAL ASPECTS OF ACCEPTABLE RISK ARE HANDLED
  • 39. VALIDATION OF INSTRUMENTS SHOULD INCLUDE: CLINICAL:  Determine specifically what your facility is going to use the test for.  Determine the environment in which the instrument will be used.  Establish clinical standards for Acceptable Risk for that facility if possible. LABORATORY:  Validate by linear regression/bias plot to establish reliability over a range of values  Validate by modified Cohen’s statistic at MDP’s to estimate our part of risk COOPERATIVE:  Determine if the instrument meets clinician perception for Acceptable Risk  If so, implement with full orientation, training, and procedural documentation  Initiate period of full monitoring to assure operational reliability  Monitor estimated technical error to identify any significant shifts  Validate clinical protocol to assure it works in THAT facility. RECOMMENDED STEPS IN ESTABLISHING POC TESTING AN INTEGRATED APPROACH BETWEEN CLINICAL AND LAB STAFF IS BEST
  • 40. Ideally, the laboratory should travel down the following path to the point where it can evaluate clinical risk with great confidence. RECOMMENDED STEPS IN VALIDATING TESTING METHODOLOGIES THIS REQUIRES AN INTEGRATED SOFTWARE APPLICATION TO ASSIST US IN VALIDATION STEP QUESTION TO ANSWER TYPE STATISTICAL TOOL RECOMMENDED* 1 Is the data unimodal? MDP Data Distribution Plot 2 Is the data normally distributed? MDP χ2-Test 3 Is estimated precision low? MDP SD CV / Range 4 Is estimated accuracy high? MDP Reference – Mean / Reference - Median 5 Are the SD’s the same? COM F-Test 6 Are the Means the same? COM T-Test 7 Is there a linear relationship? COM Regression 8 What is the range of linearity? COM Bias Plot evaluation 9 What is the overall reliability? COM Correlation Coefficient 10 What is the concordance at MDP’s? MDP Simplified Cohen’s Kappa * NOTE: There are many others in the literature.
  • 41. SUGGESTED FLOW CHART TO ASSURE CLINICAL RELIABILITY ACCEPTABLE RISK SHOULD DRIVE CLINICAL AND LAB TEST MONITORING Selection of Analyte Selection of Test Methodology Clinical Application Defined Risks for Each State Defined Clinical States Defined Acceptable Risk Defined Technical Error Limits Defined Validation of Methodology Acceptable Performance? YES NO Develop Utilization Policies Institute Adjustments if Needed Write Technical Procedures Publish Estimated Reliability Orient and Train Users Monitor Analytic Reliability Monitor Clinical Results Carry out Ongoing Upgrades Retire at End of Life Cycle Risk Distribution(s) Defined
  • 42. PUTTING IT ALL TOGETHER AN INTEGRATED APPROACH IS BEST In his 1989 article on Controlling Error in Laboratory Testing Donald Forman, Ph.D puts it all together in an integrated overview of problems which is applicable to all Clinical Laboratory Testing: “…correct interpretation of test results requires knowledge of analytical and biological as well as pathophysiological sources of variation, their expected magnitude, and the time course over which changes can occur in health and disease.” In just one sentence Dr. Forman includes all the critical factors we as laboratorians and our clinicains must consider when implementing, measuring, reporting, and applying any test analyte result. The operational and technical problems we face in doing so are very significant. Add in cultural resistance on the part of laboratorians and physicians and you have a serious obstacle to improvement in the laboratory. Forman, D.T., MLO Supplement 1989.
  • 43. THERE IS MUCH STILL TO DO: It is clear that, today despite miraculous advances in the knowledge and application of that knowledge to the clinical laboratory we are still faced with significant challenges when it comes to the appropriate validation of the tests we place in the hands of our clinical staff. KNOWLEDGE AND UNDERSTANDING IS LACKING: Presently our clinicians are not educating themselves adequately in the fundamental science underlying these tools in order to understand in detail their strengths and weaknesses and this has lead to health care by cook book and not medicine by clinical acumen placing the laboratory in a difficult situation for expected accuracy and precision. WHAT IS TO BE DONE? If we are going to maintain our position as a highly regarded profession that is dedicated to service to our clinicians and patients then we must begin to rethink how we apply all this technology to our efforts at clinical evaluation and management. Otherwise, soon, we will be submerged under a collective system of government regulations and accreditations standards that may do more harm than good. CONCLUSIONS AND WARNINGS FINIS! QUESTIONS?
  • 44. REFERENCES REFER TO THE SEPARATE REFERENCE LISTING