Neil Greenberg, PhD, DABCC, Manager, Regulatory Affairs, Ortho-Clinical Diagnostics, Inc., A Johnson & Johnson Company
This article is published with permission and originally appeared:
Neil Greenberg, "Calibrator Traceability: The Industry Impact of the IVD Directive's New Requirements," IVD Technology 7, no. 2 (2001): 18-27

Download as a PDF here

Introduction

Traceability to internationally recognized and accepted standards is an important component in assuring the accuracy and comparability of clinical laboratory measurements. Currently, the global marketplace is presenting new demands on IVD device manufacturers for measurement traceability. Under a mandate from the European Union�s In Vitro Diagnostics Directive1 (IVDD), the European Committee for Standardization�s Technical Committee 140 (CEN/TC 140 - In Vitro Diagnostic Systems), together with ISO/TC 212, is currently developing international standards on IVD calibration traceability2,3. Full implementation of the IVD Directive, which is obligatory by December 2003 under European law, will require that calibration of quantitative IVD assays be traceable to available �higher-order� reference materials or methods.  Manufacturers who implement the processes and documentation steps defined in the calibration traceability standards are entitled to a presumption of conformity to this �essential� requirement of the IVD Directive.  To do this, IVD manufacturers must ensure that the systems they market have been calibrated against available higher-order reference standards and procedures, that repeatability and reproducibility of their internal calibration procedures are quantified and documented, and that accuracy is substantiated by uncertainty calculations.

The draft international standard for IVD calibration traceability, ISO/DIS 17511, identifies the essential elements of a calibration hierarchy necessary to support full calibration traceability to defined units of measure under the Syst�me International (e.g. moles, kilograms).  In addition to a complete definition of the quantity to be measured, there is a need for primary standards, including material standards as well as standard methods of measurement.  In the introductory section of the text, the standard also discusses a key problem that exists for measurement systems in the field of in vitro diagnostics. This is that, although the in vitro diagnostics field routinely performs measurements on an estimated 400 to 600 different amounts of substances (analytes), full calibration systems with traceability to SI currently exist for less than 30 (perhaps 5%) of these analytes.  What will be done to fill this 95% void, and what role will be assumed by the IVD industry?

Do IVD manufacturers want additional international reference materials and methods to be developed?

Despite short-term concerns, international IVD calibration standards (reference systems) ultimately help IVD manufacturers by providing well-defined market needs and customer requirements, a clear and universal definition of goals, and tools for objective assessment of product attributes. Table 1 lists a few of the global trade-offs to be evaluated by all participating organizations (manufacturers and distributors of IVD products, industry associations, user groups, customer advocacy and professional groups, government and regulatory bodies) upon considering development of new calibration standards (reference systems) for IVDs. It is clearly a matter of cost vs. benefit, and the decision to invest in standardization demands careful analysis on a case-by-case basis.

TABLE 1. Decision Trade-Offs for Investment in New IVD Reference Systems

An Example: Economic Impact Analysis of Standard Reference Materials for Cholesterol:

A recent economic impact study conducted by the US National Institute of Standards and Technology (NIST) quantified a portion of the economic benefits associated with the availability of NIST Cholesterol Standard Reference Materials beginning in 19864. This study determined that the economic consequences of NIST�s Cholesterol Standards Program were experienced at several levels of the IVD medical device supply chain from manufacturers, to network laboratories, and to clinical laboratories that ultimately deliver medical services to the consumer. The nature of the benefits to industry resulting from the NIST investment have changed over more than three decades of NIST involvement. However, because the timeframe of this analysis was limited to 1986-1999, only part of the NIST program�s life cycle, the magnitude of the estimated economic impact was biased low. Nevertheless, the results indicated that NIST reference materials played an important economic role in support of the US national effort to monitor, measure, and control cholesterol levels, thereby contributing to a reduced level of heart disease. The study estimated a benefit-to-cost ratio of 4.5, and a social rate of return of 154 %. The Net Present Value was calculated to be more than (US) $3.6 million. The study did not attempt to account for the impact of NIST reference materials on reducing the incidence of incorrect laboratory measurements on patient care, which has been estimated to be approximately $100 million per year in the US.

Impact of Measurement Bias on Globalization of Healthcare

In the 21st century, we live in an era where patients frequently move from town to town or even country to country as often as every few years.  Additionally, for economic reasons, there is an increase in the rate at which patients change to different health care plans and different clinical laboratories.  As the mobility of medical records increases along with patient mobility, differences in laboratory test results among laboratories and across different test methods are becoming increasingly apparent to physicians and other health care practitioners.

Problems due to lack of consistency in standardization among analytical methods were highlighted and publicly debated in the United States during the 1980�s, following the US National Institutes of Health publication of its findings on the relationship between serum cholesterol levels and risk for cardiovascular disease. Similarly, in the mid-1990�s, a recommendation was published for monitoring the change in serum PSA over 2 to 4 years, as an aid in the diagnosis of prostate cancer. This recommendation made inter-laboratory variability in PSA determinations highly problematic, since a large change in PSA could result simply because of analytical differences in the methods used by two labs.  Alternatively, clinically significant changes in true PSA levels could be masked for the same reasons.

Klee5 has evaluated the cost-impact of certain biased laboratory test results associated with unnecessary followup of mis-classified patients. For a screening test such as serum cholesterol, patient mis-classification is likely to be followed up with expensive additional testing or even inappropriate treatment. Using actual test results distributions rather than Gaussian models, Klee estimated that an assay for serum cholesterol that is biased 1.0% high results in a 3.0 % increase in the number of patients classified as having �high� cholesterol values.  Similarly, a 3 % bias causes an 8.8 % increase in the number of patients classified as hypercholesterolemic, and a 10 % bias causes a 27.8 % increase. Overall, the change in the percentage of patients crossing the serum cholesterol decision threshold (5.17 mmol/L; 200 mg/dL) increases about three fold more than the percentage value of the analytical shift due to the multiplier effect of the distribution curves. Based on these estimates, improvements in laboratory standardization (especially for screening tests for clinical conditions with high prevalence) makes sound economic sense, whenever the outcome is likely to be lower rates of patient mis-classification.

Key factors leading to successful improvement in inter-laboratory standardization

Why do some standardization programs meet with success, while other programs seem to languish in some instances for years without yielding noticeable benefits?  Following up on the examples of cholesterol and PSA discussed above, it is clear that the programs to improve standardization of serum cholesterol measurements yielded measureable success in terms of a very substantial reduction in measurement variability among laboratories over a ten year period.  However for PSA, although some progress is evident, success remains elusive6. Similarly, inter-method and inter-laboratory standardization efforts for measurement of human chorionic gonadotropin (HCG) in serum and urine, another clinically important analyte which is an excellent marker for screening and monitoring normal and abnormal pregnancy as well as certain malignancies, have also encountered significant technical barriers7.

In a recent review of the history of standardization efforts in the clinical laboratory, Eckfeldt8 identified four significant antecedents to successful laboratory standardization programs.  These include:

1. Results of a widely publicized clinical research study conclude that clinical action based on application of uniform cut-points for a particular laboratory test leads to significant improvement in detection and prognosis of patients with disease.  This new information leads to pressure from large clinical organizations for improvements in test method accuracy and reliability.
2. A high-level reference method and/or material exists.
3. Mechanisms exist to easily and reliably disseminate the accuracy base provided by the reference method and/or material.
4. Tools exist to reliably evaluate and publicly display inter-method and inter-laboratory performance data.  EQAS programs are a primary source of such information, and the value of such programs depends on the free flow of information about their procedures and the test materials that they distribute.

The importance of the reliability of EQAS data in assessment of the accuracy of laboratory methods is often underestimated. It is essential that EQAS materials achieve the highest possible level of commutability, so as to ensure their validity in representation of performance of a given laboratory test with patient samples. Indeed, incorrect assumptions about the validity and commutability of EQAS materials may lead to incorrect conclusions and even serious errors on the part of IVD kit manufacturers as well as EQAS providers and laboratorians. In the 1980�s, before information about commutability problems and matrix effects with manufactured (typically lyophilized) serum controls and EQAS/PT materials was widely publicized, there were examples where manufacturers unwittingly adjusted the calibration of their IVD devices and reagents to make PT/EQAS samples� results comparable to reference method target values.  In some cases, these adjustments compromised accuracy with patient samples.

Concerns about the suitability of PT/EQAS materials have led to uncertainty regarding the value of PT/EQAS results in understanding the state-of-the-art for trueness.  These concerns have also been responsible for creating some tension between providers of PT/EQAS programs and IVD manufacturers whose commercial methods are evaluated by these programs. IVD manufacturers often respond that inferior PT materials do not accurately simulate clinical specimens, and mis-represent the performance of their methods.  The providers of PT/EQAS materials and programs often argue that commercial reagent and instrument systems are insufficiently "robust". In reality, the "problem" must be shared by the broader professional laboratory community and the commercial IVD industry as a whole.

Declaration of New or Improved International Reference Materials and ReferenceMethods � Some Manufacturers May Need to Change Calibrations for Certain Analytes

Development of new or improved reference systems is not a specific requirement of the IVD Directive. The Directive states that routine methods need to be traceable to ��available reference measurement procedures and/or available reference materials of a higher order.� Similarly, ISO/CD 17511 allows for a wide range of scenarios, including situations where there is no recognized higher order calibration method or material available to trace back to. Under these circumstances, the highest order reference point available to a given IVD manufacturer may be a measurement procedure or reference material that is uniquely defined, controlled, and maintained by each manufacturer of the various commercial assay systems for a given analyte. ISO/CD 17511 takes a stronger position in terms of commitment to the cause of continuous improvement in reference methods and materials. As the standard states, �It is the aim of metrology in laboratory medicine to improve traceability�by providing the missing reference measurement procedures and reference materials, based on international consensus.� Because of this implied commitment, it is expected that many national and international standards organizations, scientific, professional, and industry groups, will interpret the Directive�s intent, arriving at an interpretation which says that the Directive demands investment in upgrades to the international reference system for the clinical lab.

When new reference methods or materials are developed and become globally accredited, certain IVD manufacturers will inevitably have to make changes in their internal calibration procedures in order to adjust performance of their products to become standardized to new reference systems9. The cost implications for these changes are far-reaching, and will impact end-users as well.  Table 2 highlights a few of these costs. 

Most successful IVD device manufacturers are willing to step up and implement calibration changes necessitated by customer needs, especially when these changes are expected to be beneficial in bringing about improved health care. Hopefully, the required changes will be important ones, representing needs articulated by a broad, global consensus of clinicians and laboratory professionals, since many changes may be associated with disruption and costs. Given these costs, it is especially important that the initiative and the leadership for change be customer-focused, originating from a clear expression of need for improvement on the part of the end-users, not the manufacturers.

What is most important, as new reference materials or reference method projects are initiated in the name of the Directive, is that adequate scientific support from industry be sought when staffing the technical working groups. Active and meaningful participation by industry scientists will help to ensure that a reasonable balance is achieved between commercial interests, pragmatic realities of manufacturing materials and process limitations, and academic clinical and metrological interests.

Does Industry Support Initiatives for New IVD Calibration and Reference Systems?

Public interest requires good quality and safe health care products. It is easy to demonstrate that improved standardization contributes to furthering these goals, and all IVD manufacturing companies share this interest. Reference materials and reference methods for calibration are an important underlying element, contributing to quality and safety through the assurance of interchangeability of information across time and space.

Given these shared goals, it is inevitable that new reference systems projects will emerge, and new reference materials and methods will ultimately be defined, impacting the definition of the state of the art. Industry must play a role in this process, using its collective wisdom achieved through years of real-world experience, to ensure technically sound and practical solutions to the challenges encountered in development projects undertaken in the quest for better standards.

Recommendations

What factors are necessary to get to the future state? To begin, laboratory medicine, scientific, and professional organizations need to provide leadership and guidance relative to what standards are needed. In doing so, project priorities should be defined with an appreciation that resources are limited, while taking into account factors such as (1) the public health significance and disease course, (2) expectations of the degree of improvement anticipated in overall clinical effectiveness of a given test if a new standard is developed, (3) time and cost estimated to reach a desirable endpoint, and (4) the overall likelihood of success.

High priority should be given to establishing a defined, global, customer-focused, and consensus-based process for setting priorities and contracting projects. This process should be led by a consortium of the world�s major laboratory professional associations (e.g. IFCC, AACC, CAP, WASP and others) and should proceed in an atmosphere of open and public dialogue, employing decision tools that emphasize quantification and metrics, and is inclusive of all key stakeholders (i.e., profession, government, industry, lay public).

Sufficient project funding (grants or contracts) must be provided, and should include government sources (e.g. EU Commission, US Department of Commerce, etc.), professional societies (e.g. IFCC, AACC, CAP, WASP), as well as industry groups. Projects must be closely managed, with accountability for deliverables and schedule. Project teams must be staffed with appropriate clinical and scientific experts, coming equally from the professions and industry.

Conclusions

Although the EU IVD Directive does not specifically require it, reference materials and reference methods development projects will be initiated in the name of the Directive�s essential requirement for calibration traceability. New calibration standards are likely to increase short-term costs for IVD manufacturers, but this is a minor consideration if there is a clear need for improved standards and an expectation of improved quality of health care as an outcome, as expressed by a consensus of customers and professional associations.

Professional and customer advocacy groups should take the lead role in advocating for new calibration standards, especially in defining where improved standards are needed. Project selection must utilize cost-benefit analysis, taking into account public health payback, technological limitations, and magnitude of the investment necessary to achieve the desired outcome.

Whenever IVD calibration and reference systems standards projects are undertaken, whether sponsored by professional, government, or public health groups, inclusion of IVD industry scientists and experts on the technical team is an absolute prerequisite for project success. Creative strategies are needed to ensure adequate project funding, and should involve a combination of contributions from public, professional and industry sources.

References

1.        Council Directive 98/79/EC of the European Parliament and of the Council of 27 October 1998 on In Vitro Diagnostic Medical Devices, Official Journal of the European Union L331 (December 12, 1998).

2.        In Vitro Diagnostic Medical Devices�Measurement of Quantities in Samples of Biological Origin�Metrological Traceability of Values Assigned to Calibrators and Control Materials, ISO/CD 17511 (Geneva: International Organization for Standardization, February, 2000).

3.        In Vitro Diagnostic Medical Devices�Measurement of Quantities in Samples of Biological Origin�Metrological Traceability of Values for Catalytic Concentration of Enzymes Assigned to Calibrators and Control Materials, ISO/CD 18153 (Geneva: International Organization for Standardization, February, 2000).

4.        May, Willie E, NIST Measurement Methods and Standards Reference Materials for Health Status Markers: Current Program and Future Challenges, in: NISTIR 6742, Proceedings of the Workshop on Measurement Traceability for Clinical Laboratory Testing and In Vitro Diagnostic Test Systems, Ellyn S. Beary, Editor, May 2001, U. S. Department Of Commerce, National Institute of Standards and Technology.

5.        Klee, George G., Importance of Commutable Reference Materials and Patient Test Distributions for Assay Calibration, in: NISTIR 6742, Proceedings of the Workshop on Measurement Traceability for Clinical Laboratory Testing and In Vitro Diagnostic Test Systems, Ellyn S. Beary, Editor, May 2001, U. S. Department Of Commerce, National Institute of Standards and Technology.

6.        Chan DW, Sokoll LJ. WHO first international standard for prostate-specific antigen: the beginning of the end for assay discrepancies? Clin Chem 2000;46:1291-2.

7.        Galina Kovalevskaya, Steven Birken, Tatsu Kakuma, John Schlatterer and John F. O�Connor, Evaluation of Nicked Human Chorionic Gonadotropin Content in Clinical Specimens by a Specific Immunometric Assay, Clinical Chemistry, 1999; 45: 68-77.

8.        Eckfeldt, John H., History of Reference Systems for Clinical Measurements, in: NISTIR 6742, Proceedings of the Workshop on Measurement Traceability for Clinical Laboratory Testing and In Vitro Diagnostic Test Systems, Ellyn S. Beary, Editor, May 2001, U. S. Department Of Commerce, National Institute of Standards and Technology.

9.        Powers, Donald M., Regulations and Standards - Traceability of assay calibrators: The EU's IVD Directive raises the bar, IVD Technology, July-August, 2000.