The smart operating theatre with digital twins

All is quiet in the operating theatre. The only sound is the gentle humming of a camera on the ceiling. The surgeon is bending forward slightly, above the operating table, and in front of her is a digital twin of the patient’s spine – true to scale and floating in the air, directly above the actual body. No hectic shouting, no searching for information. The next step of the surgery is already announced in the form of discreet markings on the surgical drapes, with additional, AI-supported visual feedback: 'Incision confirmed: spinal column in desired position.'

The instruments – made of traditional metal, but fitted with sensors – send continuous data regarding position, force and tissue status to a central computer. Software analyses hand movements, eye movements and biomechanical loads in real time. The scene described is not science fiction, but a foretaste of a future that has already begun in Zurich.

SURGENT project provides the spark

In 2018, SURGENT (Surgeon Enhancing Technologies) was launched as a project involving the University of Zurich and Balgrist University Hospital. The aim was a fundamental rethink of how surgery is planned and performed – with a shift away from screens and static measurements and towards a holistic, AI-supported system. The focus was on spinal surgery, where highly precise interventions are critical to treatment success.

A total of six patients with chronic back problems, who had failed to respond to conventional therapy, were to undergo surgery as part of the project. Screws and rods were implanted with tremendous positional accuracy – supported by extended reality technology, with the related images still displayed, at the time, via mixed reality glasses.

Even during the project, it became clear that these large glasses would become redundant. In future, modern sensors and AI modules will increasingly take over the task of relaying relevant information directly to surgeons – in real time. Instead of bulky headsets, surgeons in future will wear light, unobtrusive data glasses displaying information. These glasses show things like patient data, navigation inside the body or warnings – without blocking the user’s field of vision. Making operations both safer and easier to plan.

Training in partnership with technology

The success of the first interventions paved the way for the next step: the foundation of the translational research and teaching centre, known as 'OR-X', at Balgrist University Hospital – a place where future technologies are trialled and training is provided in a realistic environment. 'At OR-X, we have moved on from confining ourselves to theory alone or the traditional "see one, do one" principle,' explains Philipp Fürnstahl. He is the scientific coordinator of the SURGENT research project, which was led by UZH professor Mazda Farshad and ETH professor Mirko Meboldt. In a medical context, the 'see one, do one' principle means first seeing how an intervention is performed – and then doing it yourself.

Sponsored by Innosuisse, the 'PROFICIENCY' follow-up project is shifting further education for surgeons towards a realistic and technologically advanced training system. Physicians train on bone models with genuine instruments, while guided by extended reality. At the same time, cameras and sensors document every physical movement, every eye movement and any deviation. The data is used to generate photorealistic 3D reconstructions – so-called digital twins – that can be analysed, replicated and compared. 'Our vision is to implement a standardised, measurable and AI-optimised approach to training,' says Fürnstahl.

Learning with simulations and detailed feedback

Meanwhile, the 'IMMERSE' project is developing an interdisciplinary ecosystem for learning. Extended reality applications, based on artificial intelligence, are set to become the norm within university-level education – and not just in medicine. Here, physicians get to experience digital simulations, combined with detailed feedback: performance is analysed, evaluated and improved – a revolution in teaching that is both data-based and immersive.

And lastly, 'AUGMEDI': the UZH start-up is creating photorealistic anatomy models for tablets, smartphones or screens. Over 3600 anatomical structures can be investigated interactively, with the opportunity to practise surgical steps virtually – enhanced by AI-supported learning analyses. Initial studies show how this is making knowledge more accessible and helping to embed it 40 per cent faster.

At the same time, new generations of surgeons are learning with digital twins, extended reality simulations and AI-controlled feedback. What started in Zurich with SURGENT is more than just a project now. It is a whole new approach to surgery, based on technological support, precision and adaptability.

Are you suffering from back or neck pain
or would you like a second opinion?
Clinic at Balgrist University Hospital

SURGENT:
The name of the project. It stands for SURGeon ENhancing Technologies. A technology that enhances the abilities of the surgeon.

Extended reality (XR) is an umbrella term for technologies that combine real and virtual environments. It covers:

• Virtual reality (VR) – a fully virtual world
• Augmented reality (AR) – digital content via the real world
• Mixed reality (MR) – real and virtual elements interacting together

Incision:

A cut made during surgery into tissue, usually with a scalpel

Text: Marita Fuchs
Audio: Rebekka Haefeli
Pictures: Tagesanzeiger, Sabina Bobst
University of Zurich: Mazda Farshad, Philipp Fürnstahl
Balgrist University Hospital: Mazda Farshad, Philipp Fürnstahl
ETH Zurich: Mirko Meboldt, Marc Pollefeys