What is augmented reality in medicine?

Augmented reality, or AR, is a relatively new technology in which a computer-generated image is superimposed on the user’s vision of the world.¹  To create this augmented reality, hardware such as headsets, smart glasses or mobile devices are used. The difference with virtual reality is that the user keeps a link to the surrounding physical world.¹

Augmented reality has many uses in medicine. These include medical training, especially anatomy but also simulation training. Surgeons can use AR to plan surgery, and all physicians can use AR to explain complex situations to patients and their relatives.²

Diagnostics

In diagnostics, AR has been used to improve adenoma detection rate. A combination of computer vision algorithms and a large database of colonoscopy polyp images means the endoscopist gets real-time visual assistance. Images are overlaid on the primary monitor they are using or on an adjacent monitor.³

Therapeutics

Therapeutics is another area where AR has been extensively used, especially in rehabilitation. The interactive aspect means that patients are encouraged to improve their motor actions.⁴ For people with severe mobility issues, including the elderly and paralyzed, AR becomes an integrated part of their daily life as part of a home appliances system. AR interacts with brain-computer interfaces to give back patients a degree of autonomy.⁵

When ultrasound was brought in, a new 2D perception of a 3D space was needed. Anyone who has ever used an ultrasound knows that this involves retraining your way of looking at spaces in what I felt was initially counterintuitive.  Ultrasound-guided biopsy is a minimally invasive procedure for tumour staging. Still, it requires long training not only on a manual technique level but also taking into account the change in perception of space. AR is used to plan the trajectory of the needle and then execute the process. A robot arm with pressure sensors is used, feeding back high-quality information to the operator. The person undertaking this ultrasound-guided biopsy is then able to overcome any needle deflection or target motion.⁶ 

Training

Anyone who has taken a basic or advanced life support course will remember meeting Resusci Annie, the rubber mannequin used to simulate emergency situations. Although a great resource for many years, there was never any doubt that you were dealing with a floppy doll. High fidelity simulation training uses complex mannequins who can breathe, have a variable heartbeat and affect ECG readings which take training to a level. The ultimate challenge is simulation training with a real person, but there you are limited to one hopefully stable pathology, and obviously, you can’t administer medications or electric shocks. When it comes to training in anatomy, there are financial, ethical and supervisory constraints on the use of cadavers.⁷

You also can’t see inside the body, and this is where AR takes medical training to a whole new level. One setting is airway training, where learning to intubate often means switching between the student and instructor who attempts to explain what they are seeing and how best to proceed with the tube. In surgery, AR laparoscopic training too has been shown to increase trainee skills, especially when combined with physical models.³ This freedom of sight is also a safety aspect.⁸ In addition, AR means the training can take place in a professional work environment, undertaking real tasks. Depending on the program used, this training can be independent without the need for an instructor to be constantly there.⁷ Emergency medicine training has already been done remotely using AR as distances can be a real issue in more remote clinical settings.⁹

There can be some disadvantages. Sometimes trainees find that AR can lead to dizziness or blurred vision, although less than with VR or virtual reality.⁷ Cost is another consideration, although this may be less important to students and institutions who see the skill gain as non-negotiable.

How soon will AR come into my practice, and how should I prepare?

Google Glass was the forerunner of easy access VR and which some considered being low-level AR. Some of you may have tried out these glasses in a non-clinical setting. Google Glass is a good entry-level AR due to the familiarity of the concept. Many of us already use normal eyeglasses. The first version is now obsolete, but the 2020 revised version has been launched with an increased facility for developers to build their own software.¹⁰ Now more than ever, as a practising clinician, if you think of a solution for an everyday frustration, you can approach developers to build it for you. The hands-free aspect in a sterile or semi sterile environment is an attractive proposition for situations where you need access to information but don’t have the staff, such as in primary care. Being able to easily scan patient records without the need to be looking at a computer all the time would in itself make a lot of patients and doctors happy.¹⁰  In the same way as AR has helped with polyp identification in real-time, external dermal or other lesions too will be superimposed with AR and the corresponding algorithms and knowledge databank.

However, machines, like humans, are not infallible and knowing where they may fall down leads to using them more safely. Although some authors claim that AR will be trained to see with fidelity and without bias, bias in algorithms is only now starting to rear its ugly head.³  There have been several high profile cases of algorithms misidentifying people of colour in facial recognition programs.¹¹ The algorithm will only ever be as good as the input data, even if the data is extensive in quantity. Humans choose the data which will be used, and we all have our own unrecognized biases. Hidden or unidentified health inequalities are often a direct result of these biases, whether race, age or other.

Physicians may be concerned with privacy issues. In cultures where scribes writing down the notes are usual practice, the idea of someone doing the same thing remotely as you use google glass or another similar device may not be a problem.¹⁰ For other clinicians, this may take a bit more getting used to. The developers need to think like a doctor, like all doctors, to overcome resistance. Perhaps some clinicians prefer to have limited options, not all of them. At least at the beginning.

What do patients think about it?

It’s very hard to know what patients think of their doctors using AR. There is a lot of information available projecting on to patients what they should be thinking and how they should see improvements. Yet this may not be the reality. We need to ask them and listen. Specific AR therapies have good outcomes as defined by the study researchers, but you don’t know what you don’t know. Perhaps dizziness may be too much of an issue, or perhaps there are other side effects or worries which have yet to be voiced. As with telemedicine, these reticences can often be overcome once the real underlying worries are identified.¹²

So what now?

AR is one more technology that will come to the patient interaction. It’s only a matter of time. Like POCUS, point of care ultrasound, there will be fans and detractors. Individual knowledge and training are the keys, as is listening to patients. Even if you don’t like it, your patient may have heard about great outcomes for their specific condition. Or you may be encouraged by the increased safe prescribing options of AR but find that you lose patient engagement, and much as the course of antibiotics is not finished, the AR stays in the box after the first couple of days.

If you’ve had any feedback or have any thoughts on VR or AR from your patients or yourself, I’d love to hear from you. @alice_bbyram on Twitter or email me abyram@ab-health-solutions.com.

References

1.        Tang, S. L., Kwoh, C. K., Teo, M. Y., Sing, N. W. & Ling, K. V. Augmented reality systems for medical applications: Improving surgical procedures by enhancing the surgeon’s “view” of the patient. IEEE Engineering in Medicine and Biology Magazine 17, 49–58 (1998).

2.        Eckert, M., Volmerg, J. S. & Friedrich, C. M. Augmented Reality in Medicine: Systematic and Bibliographic Review. JMIR mHealth and uHealth 7, (2019).

3.        Mahmud, N., Cohen, J., Tsourides, K. & Berzin, T. M. Computer vision and augmented reality in gastrointestinal endoscopy. Gastroenterology Report 3, 179–184 (2015).

4.        Yeo, S. M. et al. Effectiveness of interactive augmented reality-based telerehabilitation in patients with adhesive capsulitis: protocol for a multi-center randomized controlled trial. BMC Musculoskeletal Disorders 2021 22:1 22, 1–9 (2021).

5.        Park, S., Cha, H. S., Kwon, J., Kim, H. & Im, C. H. Development of an Online Home Appliance Control System Using Augmented Reality and an SSVEP-Based Brain-Computer Interface. 8th International Winter Conference on Brain-Computer Interface, BCI 2020 (2020) doi:10.1109/BCI48061.2020.9061633.

6.        Freschi, C. et al. Ultrasound guided robotic biopsy using augmented reality and human-robot cooperative control. Proceedings of the 31st Annual International Conference of the IEEE Engineering in Medicine and Biology Society: Engineering the Future of Biomedicine, EMBC 2009 5110–5113 (2009) doi:10.1109/IEMBS.2009.5332720.

7.        C, M., Z, Š., A, R. & A, S. The effectiveness of virtual and augmented reality in health sciences and medical anatomy. Anatomical sciences education 10, 549–559 (2017).

8.        D, P. & K, M. Current Perspectives on Augmented Reality in Medical Education: Applications, Affordances and Limitations. Advances in medical education and practice 12, 77–91 (2021).

9.        Munzer, B. W., Khan, M. M., Shipman, B. & Mahajan, P. Augmented Reality in Emergency Medicine: A Scoping Review. Journal of Medical Internet Research 21, (2019).

10.      TriHealth invests in Augmedix Inc.’s Google Glass health care venture – Cincinnati Business Courier. https://www.bizjournals.com/cincinnati/news/2016/04/25/trihealth-invests-in-groundbreaking-google-glass.html.

11.      Raji, I. D. et al. Saving Face: Investigating the ethical concerns of facial recognition auditing. AIES 2020 – Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 7, 145–151 (2020).

12.      Healthwatch England. Locked out: Digitally excluded people’s experiences of remote GP appointments. (2021).

13.      Liu, Y., Stiles, N. R. B. & Meister, M. Augmented reality powers a cognitive assistant for the blind. eLife 7, (2018).

14.      Kulkov, I., Berggren, B., Hellström, M. & Wikström, K. Navigating uncharted waters: Designing business models for virtual and augmented reality companies in the medical industry. Journal of Engineering and Technology Management 59, 101614 (2021).