A case study in solution design: De-conflicting KPIs through process innovation and automation.

When an 'ultra high throughput' PCR testing laboratory approached us for help with reducing their wastage, our team uncovered a uniquely tangled warren of clashing key performance indicators (KPIs). Our solution transformed their scientific and management procedures and brought about a shift in culture.

This lab, as with many labs set-up during the COVID-19 pandemic, was built at pace to meet the requirement for PCR testing capacity. Considerations around efficiency, cost-effectiveness and long-term viability were secondary to the capacity and Turn-Around-Time (TAT) needs of the wider healthcare system.

Some time after 'Go-Live', however, the lab's management team identified a problem. While targets for capacity and TAT were being met, a new KPI had emerged: cost-per-test. Supply of PCR testing had finally out-stripped demand, meaning that labs offering the lowest cost-per-test ratio would be the most competitive on the market.

On paper, the lab's service offering gave the lowest cost-per-test. So the management team were perplexed to find that:

1. Actual cost-per-test was almost double what they were expecting due to wastage and inefficiency.

2. Simply ordering the operation to work more efficiently made no difference to bringing down this ratio.

There was clearly a more systemic issue that needed addressing. Our team was therefore positioned to investigate, offer actionable insights and develop solutions.

We started the project with a Business Analysis (BA) exercise to investigate, in-depth, the operational systems and holistically understand the question-landscape.

There were four main sources for our initial data-gathering:

• Interviews with key stakeholders

• Process flow analysis

• Observations of the operation

• Data analytics

From these we sought to identify and understand motivations, business objectives, scientific systems and management systems to generate hypotheses for the root-cause of the discrepancy between expected and actual cost-per-test ratios. Empiricism, intellectual rigour, rationality and Occam's razor prevailed during our hypothesis testing.

What we found was:

1. Cost-per-test was calculated as a function of the lab's overheads (staffing, facility, energy, etc) added to reagents & consumables per data-point. In essence, however, the vast majority of the cost-per-test was occupied by the reagents & consumables.

2. An audit of stock burn rates showed that the usage rate of PCR master-mix reagents was close to 2 units per data point, when it should have been 1.

3. There was one stage in the end-to-end process that could be re-worked, and this was the RT-qPCR stage.

4. Re-work could happen for a number of reasons including but not limited to: equipment failure, control failure, inconclusive results and operator error.

It was clear then that large amounts of PCR master-mix were being used, and it was also clear that one reason it might happen was because of re-work. A question emerged: why was there so much being wasted?

Even taking into account dead volumes, expected operator error rates, and the validated assay failure rate it was too high. Taking a closer look at the process yielded the following insights:

1. PCR master-mix was prepared manually with hand-pipettes.

2. Large batches (sufficient for > 6000 tests-worth!) of PCR master-mix were being prepared in one go.

3. Batch preparation could take up to 3 h by the time operators had defrosted reagents, calibrated pipettes, and logged activity in the Laboratory Information Management System (LIMS).

4. As batch preparation took so long, both under-supply of PCR master-mix and on-demand production risked long delays. To meet their overriding embedded KPIs of TAT and capacity, operations management were incentivised to: a) produce batches based on forecasted demand; b) over-produce PCR master-mix.

Taken together with operator error, these observations manifested as an operational system that was prone to wastage, with the potential for very expensive mistakes.

In addition, the PCR master-mix records in LIMS were being produced in a non-contemporaneous fashion (the LIMS was designed under the assumption that PCR master-mix was prepared on-demand, not batches based on forecasted demand). This finding was not compliant with the ALCOA+ principles that ensure data integrity within the life sciences sector.

Knowing this, it was obvious that simply telling the operation to waste less was not going to work as minor changes like making smaller batches or on-demand production conflicted with the established culture that prioritised turn around time.

However, the requirements for TAT, capacity and cost efficiency still had to be met and in concert these presented an exquisite problem. Our team therefore set about designing an elegant solution that could satisfy all stakeholders.

Re-work was clearly the culprit when it came to an inflated cost-per-test. There was not very much that we could do within a reasonable timeframe to improve re-work due to equipment, control, or assay failure rates - but we could make improvements to the technical and management systems in place to reduce or eliminate re-work due to operator errors and halt the over-production of PCR master-mix.

We gathered and validated our requirements with our key stakeholders in the operation. These teams included leads in the operation, health & safety, facilities management, equipment management, and the supply chain. It was critical to obtain their 'buy-in' and ownership of the solution through early engagement as whatever we produced had to be ultimately useful to the lab as a whole. We facilitated a simple MoSCoW analysis with our stakeholders to decide what requirements would be included in version 1.0 of the solution, the results of which are summarised below:

• The solution must give equivalent results quality to a trained, competent operator using the existing method.

• The solution must reduce wastage of reagents overall.

• The solution must permit on-demand PCR master-mix creation.

• The solution must be within a set budget.

• The solution must meet health & safety requirements.

• The solution should be automated to eliminate the potential for operator errors.

• The solution could improve data integrity through compliance with ALCOA+ principles.

• The solution would in the future be able to keep track of left over reagents and reduce/eliminate wastage due to dead volumes.

To stay within budget, we considered 3 different robotics platforms that were already on-site and available for re-purposing. A scoring scheme appraised the suitability of each of the available platforms taking into account their costs, benefits, underlying liquid handling technology, potential for integration, traceability and time to deployment. One platform was a clear leader amongst the others.

Our design would therefore use the quality control, liquid handling and traceability features of the platform to deliver a solution that met our requirements. Crucially, it would give consistent on-demand PCR master-mix preparation: taking only 15 minutes from thaw to dispense!

We had been commissioned to carry out the Business Analysis - from this we produced a root-cause, and a solution design. These were presented back to the key operational stakeholders for validation before being shown to senior management for approval, budget sign-off and a commitment to development.

Having achieved the necessary approvals from and set the expectations of the lab's governance, we could set to work building the solution. For laboratory automation solutions in live-service environments we implement an Agile methodology to control solution release.

A project management layer oversees the staged release of a solution from development/testing to operations. It keeps all stakeholders informed of progress, analyses risk and provides administration. Development and Test functions work collaboratively to cyclically build and quality control the product. A controlled release only occurs once a solution is tested and validated, and there are sufficient trained operators to deliver the service. A hypercare package is supplied to ensure smooth deployment, knowledge and skill transfer and rapid resolution of problems.

We adapt our methodology to both the solution design and the structures of the organisation we're working with, but the underlying principles stay relatively consistent. We aim to administer change in a safe and highly controlled fashion. Stakeholders are kept informed. Solutions are developed and quality controlled before validation.

Development and Testing of a solution is, in our experience, usually the most straightforward part of a project. Where challenges arise is in the operationalisation of a solution, particularly at a large scale. Some operations run shift work, while others have only a very loose structure. Some organisations have dedicated training administration teams, while others rely on their operational staff to manage the knowledge-reservoir. Careful planning, to integrate the operationalisation into the development/testing cycle, is critical to avoid disrupting the live service, up-skill staff, set expectations and meet timeline targets.

A bespoke strategy is always required to ensure safe operationalisation, the cut-and-paste approach just isn't good enough for a smooth delivery.

This is why we engage with stakeholders at all levels; we ensure clear lines of communication are maintained; and we document and broadcast plans, assumptions, and risks to the appropriate stakeholders.

While there was pressure to bring in the solution as soon as it was ready, we had to take a risk-averse approach to the deployment as there were three key challenges that posed existential risks to the project:

• The lab's operational structure ran a day and a night shift, both on a 4-on-4-off pattern - so training staff in operation of the solution required a detailed plan for out-of-hours working.

• On-going construction work with contractors requiring access to labs meant disruption was frequent and occasionally ad hoc. We set-up new reporting lines to help us work around each other.

• Shortly before deployment a wide-spread PCR amplicon contamination event required clean-up.

To manage these risks, we applied a high level of choreographic control to the deployment with go/no-go checkpoints to minimise disruption to live systems, keep operators and contractors safe, and at all times ensure data integrity. At any point, the option to abort the deployment and roll-back existed.

We followed-up the deployment with a period of hypercare, which included on-call support and supervision. Over a one week period the usage rate of PCR master-mix reagents dropped from close to 2, to just over 1 units per data point, where it remained until retirement of the COVID-19 testing service. The operation were able to continue to meet their TAT and capacity requirements while also bringing down cost-per-test by 32% making the service highly competitive. The operational management of the automated solution was far more straightforward than the original manual method as the risk of expensive mistakes associated with production of large batches to forecasted demand had been eliminated. In the long-run, solutions we've deployed, such as the scientific and managerial transformation described here have saved £ millions in otherwise wasted reagents.

Organisations looking to implement change in a live service environment need a highly detailed and cautious plan to minimise the risk of disruption. Satisfying the expectations of many diverse stakeholders is always a challenge. Fortunately, we had the backing and buy-in of a great project team because we invested early on in developing those relationships.

The Development and Validation of the MasterMix Automation method has now been published as a pre-print, available on medRxiv here.