In the operating theatre

The smart operating theatre with digital twins

In Zurich, surgery is undergoing a rethink: with the project known as SURGENT, researchers from the University of Zurich, Balgrist University Hospital and ETH Zurich are combining artificial intelligence (AI), sensors and simulation. And in doing so, they are revolutionising both interventions themselves and the training of future surgeons.

Contact

Professor Mazda Farshad MD
Medical Director and Director
of the Spine Center at
Balgrist University Hospital
Professor in the Faculty of Medicine
of the University of Zurich
+41 44 386 12 65
E-Mail

UMZH institutions

University of Zurich
ETH Zurich
University Hospital Zurich
Balgrist University Hospital

Team

>

10

All is quiet in the operating theatre. The only sound is the gentle humming of a camera on the ceiling. The surgeon is bending forward slightly, above the operating table, and in front of her is a digital twin of the patient’s spine – true to scale and floating in the air, directly above the actual body. No hectic shouting, no searching for information. The next step of the surgery is already announced in the form of discreet markings on the surgical drapes, with additional, AI-supported visual feedback: 'Incision confirmed: spinal column in desired position.'

The instruments – made of traditional metal, but fitted with sensors – send continuous data regarding position, force and tissue status to a central computer. Software analyses hand movements, eye movements and biomechanical loads in real time. The scene described is not science fiction, but a foretaste of a future that has already begun in Zurich.

SURGENT project provides the spark

In 2018, SURGENT (Surgeon Enhancing Technologies) was launched as a project involving the University of Zurich and Balgrist University Hospital. The aim was a fundamental rethink of how surgery is planned and performed – with a shift away from screens and static measurements and towards a holistic, AI-supported system. The focus was on spinal surgery, where highly precise interventions are critical to treatment success.

A total of six patients with chronic back problems, who had failed to respond to conventional therapy, were to undergo surgery as part of the project. Screws and rods were implanted with tremendous positional accuracy – supported by extended reality technology, with the related images still displayed, at the time, via mixed reality glasses.

Even during the project, it became clear that these large glasses would become redundant. In future, modern sensors and AI modules will increasingly take over the task of relaying relevant information directly to surgeons – in real time. Instead of bulky headsets, surgeons in future will wear light, unobtrusive data glasses displaying information. These glasses show things like patient data, navigation inside the body or warnings – without blocking the user’s field of vision. Making operations both safer and easier to plan.

During spinal surgery at Balgrist University Hospital

Professor Mazda Farshad carries out around 300 to 400 spinal operations himself each year. In total, 1,200 of these operations are performed annually at Balgrist University Hospital.

Training in partnership with technology

The success of the first interventions paved the way for the next step: the foundation of the translational research and teaching centre, known as 'OR-X', at Balgrist University Hospital – a place where future technologies are trialled and training is provided in a realistic environment. 'At OR-X, we have moved on from confining ourselves to theory alone or the traditional "see one, do one" principle,' explains Philipp Fürnstahl. He is the scientific coordinator of the SURGENT research project, which was led by UZH professor Mazda Farshad and ETH professor Mirko Meboldt. In a medical context, the 'see one, do one' principle means first seeing how an intervention is performed – and then doing it yourself.

Sponsored by Innosuisse, the 'PROFICIENCY' follow-up project is shifting further education for surgeons towards a realistic and technologically advanced training system. Physicians train on bone models with genuine instruments, while guided by extended reality. At the same time, cameras and sensors document every physical movement, every eye movement and any deviation. The data is used to generate photorealistic 3D reconstructions – so-called digital twins – that can be analysed, replicated and compared. 'Our vision is to implement a standardised, measurable and AI-optimised approach to training,' says Fürnstahl.

Learning with simulations and detailed feedback

Meanwhile, the 'IMMERSE' project is developing an interdisciplinary ecosystem for learning. Extended reality applications, based on artificial intelligence, are set to become the norm within university-level education – and not just in medicine. Here, physicians get to experience digital simulations, combined with detailed feedback: performance is analysed, evaluated and improved – a revolution in teaching that is both data-based and immersive.

And lastly, 'AUGMEDI': the UZH start-up is creating photorealistic anatomy models for tablets, smartphones or screens. Over 3600 anatomical structures can be investigated interactively, with the opportunity to practise surgical steps virtually – enhanced by AI-supported learning analyses. Initial studies show how this is making knowledge more accessible and helping to embed it 40 per cent faster.

At the same time, new generations of surgeons are learning with digital twins, extended reality simulations and AI-controlled feedback. What started in Zurich with SURGENT is more than just a project now. It is a whole new approach to surgery, based on technological support, precision and adaptability.

In conversation with…

Listen

Mazda Farshad (Audio file in German, autumn 2021)

«The technology allows us to personalize our treatments»

Professor Mazda Farshad MD is Senior Physician for Spinal Surgery and Orthopedics, Director of the Spine Center at Balgrist University Hospital, and professor in the Faculty of Medicine of the University of Zurich.

Mazda Farshad
7:41

Mirko Meboldt (Audio file in German, autumn 2021)

«Our job is to improve the interaction between human and machine»

Professor Mirko Meboldt is Professor of Product Development and Design at ETH Zurich.

6:09

Philipp Fürnstahl (Audio file in German, autumn 2021)

«We are giving the surgeons X-ray vision»

Professor Philipp Fürnstahl is Professor of Orthopedic Research specializing in the application of computer technologies and head of the ROCS team at Balgrist University Hospital.

Philipp Fürnstahl
6:01
In the operating theater with Mazda Farshad

Watch

Technology and innovation for better patient care

3:33
From research to clinical practice

Service

Are you suffering from back or neck pain
or would you like a second opinion?
Clinic at Balgrist University Hospital

Important terms explained in brief

Glossary

SURGENT:
The name of the project. It stands for SURGeon ENhancing Technologies. A technology that enhances the abilities of the surgeon.

Extended reality (XR) is an umbrella term for technologies that combine real and virtual environments. It covers:

  • Virtual reality (VR) – a fully virtual world
  • Augmented reality (AR) – digital content via the real world
  • Mixed reality (MR) – real and virtual elements interacting together

Incision:

A cut made during surgery into tissue, usually with a scalpel

Credits

Text: Marita Fuchs
Audio: Rebekka Haefeli
Pictures: Tagesanzeiger, Sabina Bobst
University of Zurich: Mazda Farshad, Philipp Fürnstahl
Balgrist University Hospital: Mazda Farshad, Philipp Fürnstahl
ETH Zurich: Mirko Meboldt, Marc Pollefeys

previousnext