Wayne State University researchers advance the use of AI in surgery

DETROIT - Wayne State University researchers are exploring the use of artificial intelligence tools to better serve both patients and medical personnel during surgery.

Dr. Abhilash Pandya, professor of electrical and computer engineering in the James and Patricia Anderson College of Engineering, with joint appointments in biomedical engineering and the department of surgery, recently published his findings, "An AI Approach to Markerless Augmented Reality in Surgical Robots," in a recent edition of Robotics. His paper describes his work on how AI tools can use augmented reality (AR) during surgical procedures.

Collaborators on the paper included former students in Pandya's lab, Abhishek Shankar (now at Mathworks Inc.) and Luay Jawad(now at DCS Corporation).

"Augmented reality is a technique that allows surgeons to project information into the surgical scene," said Pandya. "If a vessel is blocked from view, images can be projected via augmented reality to place objects into the scene accurately. For example, it allows a CT scan to be projected onto the body of a patient during surgery."

While others have explored the use of augmented reality during medical procedures before, Pandya says his team is finding new ways via AI to make it far more accurate.

"Our work uses the DaVinci Surgical System to improve on AR accuracy," said Pandya. "We showed that using AI improves the accuracy of what is projected. In traditional approaches, you do what is called the forward kinematics process, which uses the camera system to calibrate the 3D point to project it into the scene. It has a lot of parameters that can cause inaccuracy. We trained the AI to simplify this process to make it more accurate. If you nick the wrong vessel, you could cause major vascular or pulmonary problems. By improving what surgeons see, it will make surgeries far safer."

Pandya and his team also are exploring the use of ChatGPT in combination with medical tools so those in surgery can perform tasks and look up information simply using their voice.

"ChatGPT adds a voice interface into the DaVinci system," said Pandya. "It has a camera system that knows where to look, and the surgeon can activate several things using just their voice. A surgeon can talk to the system and have the system watch both the surgeon and patient. It should be able to tell him if bleeding is happening in the patient or if the surgeon is becoming fatigued. It can track tools. We are working to build a system that is like a partner during surgery."

Pandya believes that combining AI resources into the surgical field is a natural progression and could result in numerous benefits.

"The technology is heading its way into surgery rooms," he said. "These systems will make it so surgery is safer, surgeons are better informed, and if anomalies occur, it will alert the surgical team. It also has potential applications in the teaching arena. If there were an anomaly, it creates a sort of black box like you have on an airplane. If there are any errors during surgery, you can go back and see if it was a human error, a system mistake, a problem with the tools and so forth. It will make surgeons better, it will help train surgeons, and it will make surgeries safer."

To view the article, visit https://www.mdpi.com/2218-6581/14/7/99.

To view the prototype, visit:

https://www.youtube.com/watch?v=UZa7xCtOYT0&pp=ygUIY2FyZXNsYWI%3D

# # #

Wayne State University is one of the nation's pre-eminent public research universities in an urban setting. Through its multidisciplinary approach to research and education, and its ongoing collaboration with government, industry and other institutions, the university seeks to enhance economic growth and improve the quality of life in the city of Detroit, state of Michigan and throughout the world. For more information about research at Wayne State University, visit research.wayne.edu .