Industry 4.0 technologies that once remotely guided the Mars Rover millions of miles from Earth are now being used to create a virtual reality platform to remotely train medical students and transform how surgery is observed, in a project between a machine vision company and the University of Sheffield Advanced Manufacturing Research Centre (AMRC).

“The Covid-19 pandemic has highlighted the global need for a digitally-connected world that allows for seamless remote working conditions. The ability for senior surgeons and doctors to remotely view and be immersed in a virtual representation of an operation will have a game-changing impact,” said Dr Ben Crutchley, Senior Machine Vision Engineer at i3D robotics (i3Dr).

i3Dr worked with engineers at the AMRC, part of the High Value Manufacturing (HVM) Catapult, on a bid for funding from Innovate UK for the Stereo Theatre project. The money, part of the Fast Start competition, is a share of a £40m government investment to advance innovative projects which tackle economic and societal issues resulting from the coronavirus crisis.

Stereo Theatre will build on the work already done by the Medical AMRC in Rotherham where an Industry 4.0 Digital Operating Theatre proof of concept has been built. The demonstrator combines a virtual reality (VR) digital twin, projection mapping and smart tools that enables the position of objects and clinicians to be accurately tracked in the theatre space, with relevant information displayed digitally using screens, projections and augmented reality (AR) devices.

The ability for senior surgeons and doctors to remotely view and be immersed in a virtual representation of an operation will have a game-changing impact

Ben said: “Currently, the AMRC’s Digital Operating Theatre can provide a virtual representation of the real-world theatre and monitor movements with commercial off-the-shelf (COTS) sensors and smart tools but does not have the ability to produce updated 3D models of patients. Stereo Theatre will fill this gap, allow for further upscaling of the technology and offers a revolutionary approach to both the teaching of medical students and surgical procedures.”

Both i3Dr and the AMRC believe the technology could transform surgery and teaching by enhancing the way medical students can learn remotely, performing simulated surgeries on a virtual reality training platform. The Stereo Theatre project also enables senior consultants to be engaged in surgical procedures conducted by junior colleagues remotely, meaning more patients being observed by experienced surgeons.

Digital Design Manager in the AMRC’s Design and Prototyping Group, David King, said: “When the AMRC started the Digital Operating Theatre project our vision was to develop a fully immersive real-time digital twin of the surgical environment that would be as realistic as possible and allow the possibility for remote clinical collaboration and training.

“The integration of i3D robotics real-time stereo mapping technology within the AMRC Digital Operating Theatre brings us closer to our vision and provides a solution to the need for accurate real-time 3D visualisation of patients within the digital twin environment.”

Machine vision company i3Dr, based in Tonbridge, specialises in producing hardware and software solutions for industrial environments such as steel, nuclear and aerospace. i3Dr’s core technology, stereo vision, has evolved through decades of development for the Mars Rover missions.

The aligned vision of the AMRC and i3Dr with our combined knowledge and experience makes us confident this is the start of a long and exciting partnership.

“i3Dr’s sister company, IS-Instruments, is also developing technology to address the challenges faced by the Covid-19 pandemic. Our fluorescent imaging device to determine contamination levels could also further develop the Digital Operating Theatre concept,” said Ben.

The Stereo Theatre simulator will be tested in the Medical AMRC’s Digital Operating Theatre demonstrator to prove the technology can monitor the real-world patient to update a digital twin displayed in VR and AR in real or near-real-time.

“The AMRC will provide the requirements for the 3D vision system to be installed in the Digital Operating Theatre and i3Dr will use this information to produce a vision system capable of capturing 3D data of the chest of a patient,” said Ben.

“We will further develop software to allow this 3D model to incorporate into the AMRC’s digital twin of the theatre. The system and software will finally be tested at the AMRC with the view that it may act as a demonstrator beyond the end of the project.”

David added: “This is our first collaboration with i3Dr and we are already in discussions about significantly larger new programmes. The aligned vision of the AMRC and i3Dr with our combined knowledge and experience makes us confident this is the start of a long and exciting partnership.

“The technology developed and demonstrated within the AMRC’s Digital Operating Theatre is translatable across sectors with AMRC partners currently investigating new applications within the manufacturing environment.”

The Fast Start competition aims to fast-track the development of innovations born out of the coronavirus crisis while supporting the UK’s next generation of cutting-edge start-ups.

Business Secretary, Alok Sharma MP, said: “The coronavirus crisis has created challenges that impact the way we live, work and travel but has also prompted a wave of new innovations as businesses look at ways to solve some of the challenges facing our world today.

“This funding will support UK start-ups to deliver potential solutions, services and ways of working and help ensure the long term sustainability of these businesses.”