Virtual Reality Transforming Brain Surgery for Patients and Providers

When Dr. Kurtis Auguste first tried out a pair of virtual reality (VR) goggles in 2017 to play video games with his son, he instantly understood how the immersive 3D world could apply to him. It wasn't the now-ubiquitous battle games and space travel adventures that first came to mind but a realm he was intimately familiar with: brain surgery.

"I wondered if we could use this technology to take us inside patients' brains," said Auguste, a pediatric neurosurgeon at UCSF Benioff Children's Hospitals. The question led him to consider a range of uses, including preparing for complex surgeries and inviting patients and families to take a "VR flight" – a virtual journey into the patient's own brain – to better understand their diagnosis.

Auguste was keenly aware of the limitations of two-dimensional X-ray, CT and MRI images when it came to preparing a surgical path. Two-dimensional images of the brain provide only a single view or flat slice, often leaving surgeons to face unexpected obstacles when operating deeper than the images portray. Auguste knew that an accurately scaled image of what surgeons would actually encounter could both improve the ability to plan ahead for such problems and help surgeons train to perform complicated procedures.

Seven years later, Auguste has led VR flights for more than 100 patients and their families at both UCSF children's hospitals. The technology is used for some of the toughest cases, mainly patients with brain lesions or tumors. Auguste and his VR assistant, employed by the VR console maker, meld about 300 2D images of each patient's brain – taken from every angle of the head – into an accurately scaled, virtual 3D image. This provides a road map for identifying the best surgical entry point and a clear view of any physical obstacles that could result in avoidable bleeding or damaged nerves.

"Not only is VR visually stunning, compared to bland black-and-white 2D images, but I can actually move around the 3D space and study the environment that I'm about to explore surgically," Auguste said. "You see so much more in 3D than in 2D."

A journey into the brain for patients and families

Emri Bullock was 11 years old when Auguste performed her first brain surgery at UCSF Benioff Children's Hospital Oakland. At age 4, she had been diagnosed with neurofibromatosis type 1 (NF1), a progressive genetic condition that primarily affects the skin, nervous system and eyes. Symptoms of NF1 range from mild to severe. For Bullock, the first tumor appeared in her brain.

Auguste removed the tumor, which was the size of a golf ball, in 2014, but it recurred four years later. Two more tumor-removal surgeries appeared to be successful but the areas later developed cancerous tumors, discovered on her quarterly MRI checkups. Emri consequently received intensive chemotherapy, missing several months of high school.

"Each reoccurrence was frustrating, but Dr. Auguste kept me positive, and I knew I was in good hands," said Emri, who is now 21 years old.

For Emri's mother, Jackie Bullock, having VR technology at the start of her daughter's surgeries would have made a world of difference.

"VR technology was not available when Emri first began treatment for her brain tumors," Jackie said. "If it had been, it may have considerably reduced the stress of the surgeries. When Dr. Auguste led Emri and me on a flight of Emri's brain following her last brain surgery, I felt included and relaxed because I could see it for myself and was confident that he would know what to expect if a similar tumor reoccurred."

Emri found the VR flight jaw-dropping. Following avatars of her parents and herself on a VR journey through her own brain – guided by an avatar of Auguste in a white lab coat – was unreal, she said.

"I took flights of my brain after my last surgery and couldn't believe how big the mass was," she said. "At the end, I said 'Wow, that was in my head?' But it felt great to see it and see for myself that it was gone."

For Auguste, VR helps him communicate with his patients in a venue they can relate to.

"These kids are well-schooled in video games, so they light up at wearing goggles for a flight into their own brain," Auguste said. "Seeing a physical representation of the lesion that has been causing their symptoms is very empowering. It is also vivid, colorful and visually engaging, which they love."

X-ray vision through augmented reality

As technology evolves, so does this surgeon's approach to his craft. While VR offered a computer-generated simulation of an alternate world – in this case, a representation of a patient's brain that Auguste could enter and explore to chart the safest surgical path – he has embarked on the technology's next generation by adopting augmented reality (AR) in his operating rooms.

AR blends the virtual content created from 2D images with the real world by superimposing holograms on the subject, which is then viewed with AR glasses or a mobile phone.

The result, he said, allows him to visualize a surgery with more confidence and gives him greater precision in removing abnormal tissue while preserving, avoiding and protecting the rest of the brain.

"With AR, 3D holograms are superimposed onto the patient's brain while I operate under the microscope," Auguste said. "The effect is like having X-ray vision. I'm seeing the tumor that is beneath the surface, so I can pinpoint exactly where I need to make incisions."