Technology which could ‘advance patient care and improve patient safety’ during surgery, has been developed at the University of Sheffield’s Advanced Manufacturing Research Centre (AMRC).

Project engineers Sam Hyde, Jonathan Gomez, Liam Overett, Ross Eaves and Rob Stacey, from the Design and Prototyping Group (DPG), under the guidance of Digital Design Manager David King and AMRC Medical Lead Malcolm Brown, have created a Digital Operating Theatre proof of concept demonstrator, with the help of surgical teams. It combines a virtual reality digital twin, projection mapping and smart tools, allowing the position of objects and clinicians to be accurately tracked in the theatre space, with relevant information displayed digitally using screens, projections and augmented reality devices.

Nicholas Lee, Consultant Oral and Maxillofacial Surgeon at Sheffield Teaching Hospitals NHS Foundation Trust, said: “The technology used is very innovative. Concepts like smart tooling, the virtual theatre and virtual reality are all new to me.

“I really feel very strongly that the Digital Operating Theatre has the potential to improve the environment staff work within, advance the care we provide to our patients, aid teaching and training but, most importantly, has the potential to improve patient safety.”

Sam Hyde said: “It’s the same kind of technology they use for CGI movies. Retroreflective markers are placed on clinicians and objects then the camera tracking system tracks each of these points down to 0.2mm and updates the position 120 times a second.

“Everything in the Digital Operating Theatre demonstrator is replicated in the virtual reality on a one-to-one scale, with 100% accuracy, so we can position an avatar or surgical device in real time.

“The virtual reality allows someone to view the theatre from any angle with no physical restraints. During a surgery you can look over the surgeon’s shoulder to see exactly how they are doing what they are doing. You could call up an expert on the other side of the world to provide advice and help you when you are in the middle of a surgical procedure.”

Currently, the Digital Operating Theatre demonstrator has a two-way voice communication system in place to allow those in the theatre to speak to others remotely. “The next stage of that”, said Sam, “which is being worked on at the minute, is allowing that remote person to draw or place other visual representations which then can be projected in the real theatre in real time.

“The remote person could point to something on the patient or bring up information or videos for the surgeon to see.

“Projection is, hands down, the biggest impact out of anything. Projection mapping is used at Factory 2050 where they use this kind of technology for work instructions, it just hasn’t been done in this set up and it received such good feedback from the clinicians we worked with.”

It’s hoped the Digital Operating Theatre could help overcome the physical restraints of teaching medical students, as Sam explained: “In a real surgery scenario, you can only have so many bodies in the room and they have to stand well back. With virtual reality, it means you don’t have any physical limitations. You could have 100 people looking over the surgeon’s shoulder, or even in front of him, viewing the surgery in real time; it would be physically impossible in any other way.”

Head of the AMRC Design and Prototyping Group, Craig Roberts, said: “In the main, we have modified and augmented commercially available, off-the-shelf equipment. The clever bit is with the software development and what the team have done with existing technology. It is horizontal innovation.

“This is proof of concept development at the moment. We want companies in the healthcare sector to see what we are capable of creating, such as the Smart Surgical Instrument prototypes, and to work with us to develop these range of technologies further.”

The Smart Tools developed by DPG have a digital twin concept, creating a full metric of the instrument so the virtual reality can recognise precisely where it is, the angle it is at and how the clinician is using it.

Sam said: “Rather than retroreflective markers, the tool is tracked using infrared LEDs and rotary encoders.

“All the data is being captured so we can watch it in real time or record it and play it back at a later date. As it is used in the Digital Operating Theatre, we have a virtual reality headset and avatar as well so we can view all this through the eyes of the surgeon.

Craig added: “We have asked what the requirements are of surgeons and clinicians and used their feedback to direct how we work. So far we have received very positive feedback.

“One of the things the surgeons said to us is that, on occasions, they will use a tool but it might break or the battery might run out. If you have a tool that is intelligent enough to tell them it is faulty or low on battery, surgeons will know not to use it.”

With the first phase of the Digital Operating Theatre demonstrator complete, further phases will now investigate refined tooling, track and trace technology, robotics and machine learning.

The project demonstrates the AMRC’s ability to innovate in an economical manner, as Craig said: “This has been funded through the High Value Manufacturing Catapult (HVMC), of which the AMRC is a part, and has been delivered with just a small team of engineers, in a relatively short timeframe of just 18 months.”