Summarising and contextualising the examination of scientific service performance

Article 2 in the Assay Quality Assurance Series

In the last article we established that Trust is the reason why one must find a way of showing that a service is fit-for-purpose. Let us now consider what Validation, Verification, Evaluation and Qualification mean in the diagnostics context.

We can group these terms into two categories:

Processes and Methods undergo:

• Evaluation

• Verification

• Validation

Equipment undergoes:

• Installation Qualification

• Operational Qualification

• Performance Qualification

Firstly consider Processes and Methods:

Processes and Methods can be the entire end-to-end system implemented from sample collection through to result delivery. It could also refer to individual steps in a larger multi-stage system. The scope of the Process or Method under investigation should be defined during planning.

Evaluation is a generic term for the act of comparing the performances of equivalent Processes or Methods. For instance, when comparing commercially produced kits that detect a pathogen or a biomarker to decide which kit to procure, a project team should conduct an Evaluation. Such an Evaluation might consider performance criteria such as sensitivity, specificity, accuracy and precision, but may also consider ease of use alongside value for money. Thorough Evaluations support the Business Case for proceeding with one supplier over another - and would be a key Scientific facet of a Competitive Tendering Exercise.

Validation is the examination of the entire Process or Method to confirm that its performance meets the requirements for the intended use. This is an in-depth exercise, which ensures that results are correct and consistent. Performance characteristics such as sensitivity, specificity, accuracy, precision, linearity (where appropriate), uncertainty of measurement and reproducibility are carefully tested for compliance against Acceptance Criteria. Acceptance Criteria should be carefully set to ensure that the implemented system satisfies requirements set by clients and regulators. For instance, a Validation is performed when deploying a non-standard or customised Process or Method. It is also performed when significantly altering the operational parameters of a Process or Method already in routine use (e.g. substituting a control for another product; changing the reaction volume). These would be considered to be changes that could alter performance characteristics.

Hence, Validation is required to confirm that the system performance continues to be fit-for-use. Note well that Validation is, strictly speaking, NOT a range-finding exercise. Operating parameters for a system should be determined during Research & Development. The Validation confirms that performance falls within operating parameters.

Verification is again the confirmation that an entire Process or Method meets the requirements for the intended use. However, the critical difference compared to a Validation is that a Verification is performed where a laboratory wishes to implement a previously Validated Process or Method - whether Validated by the manufacturer or at another laboratory to confirm that acceptable results can be achieved and requirements are fulfilled. Small modifications to an in-operation Process or Method such as equipment substitution would also be subjected to a Verification. Performance characteristics assessed in a Verification are very similar to a Validation, and at a minimum should include accuracy, precision and linearity (where appropriate).

Regarding Equipment:

Suppliers of analysers, cold storage and other scientific equipment typically offer installation and qualification services. Purchase of these services should be considered where appropriate, and this will depend on an organisation's regulatory stance. Qualification of equipment is formal testing and documentation that said equipment is correctly installed, configured, operating within manufacturer's specification, and performing correctly in simulated 'real' conditions.

Installation Qualification (IQ) confirms suitable installation and configuration of a system, as defined by the supplier for the specified application. For example, a liquid handler may require a 96-channel pipetting head, whereas the standard is for it to be supplied with 8-channels only. Documentation should be produced that confirms that this 96-channel pipetting head is fitted correctly, and that the instrument software is correctly configured to make use of the 96-channel head. IQ protocols also cover basic functionality such as cables, connectivity, positioning, switches and software version numbers. The scope of an IQ is specific to the piece of equipment.

Operational Qualification (OQ) confirms that a system functions correctly throughout the full range of its specification. For a liquid handler, this might be a set of test cases that verify pipetting accuracy at the upper and lower volume extremes as well as tests in the middle of the volume range. Similar principles could be applied to spectroscopic analysers and balances. OQ also concerns the use of the software that supports equipment operation, for instance, do forms accept appropriate data. An OQ protocol may also test whether technological and engineering controls that ensure compliance function as intended (for example user-groups with different permissions or lab-ware inserts that restrict plasticware types or orientations).

Performance Qualification (PQ) verifies that a system performs as expected under simulated 'real' conditions. It tests whether the implemented system meets requirements in a User Requirements Specification. Often, the end-user executes the PQ, which may take the format of end-to-end system runs (in a multi-stage workflow), load testing or testing of control material to confirm that the system behaves as expected and required. PQs may be carried-out in-house, or out-sourced, but are not usually offered by equipment suppliers.

Conclusion

Where appropriate, laboratories undertake the above assessment activities to fulfil obligations to regulators, customers and end-users. There are some shared themes and principles that are common to all of the activities, for instance - acceptance criteria and user requirements which always need to be considered. It is critical that a business conducts appropriate due diligence of its products, systems and services, because that expectation is the foundation of trust.