Open Access

Surgery for Rectal Cancer With Mixed Reality of 3D Model During Operation: A Case Report


1Osaka University, Graduate School of Medicine, Department of Gastroenterological Surgery, Osaka, Japan

2Osaka International Cancer Institute, Department of Innovative Oncology Research and Regenerative Medicine, Osaka, Japan

3Monash University, Central Clinical School, Melbourne, Australia

Cancer Diagnosis & Prognosis Sep-Oct; 3(5): 597-600 DOI: 10.21873/cdp.10260
Received 04 June 2023 | Revised 21 July 2024 | Accepted 05 July 2023
Corresponding author
Norikatsu Miyoshi, MD, Ph.D., FICS, FACS, FASCRS, Osaka University, Graduate School of Medicine, Department of Gastroenterological Surgery, Yamadaoka, Suita, Japan. Tel: +81 668793251, Fax: +81 668793259, email:


Background/Aim: Recently, robotic surgery for rectal cancer has become a common minimally invasive surgery. In addition, the technology of augmented and mixed reality is applied in various living environments, including medicine. We successfully performed robotic surgery for rectal cancer with three-dimensional (3D) images as mixed reality (MR) using HoloLens2. Case Report: The patient was diagnosed with rectal cancer by colonoscopy and a positron-emission computed-tomography scan, and we performed robot-assisted anterior resection. The operator used HoloLens2 and performed the surgery while visualizing 3D images of pelvic anatomy with the location of the rectal cancer as hologram. The operation was performed completely and safely, and she was discharged 11 days after surgery with no postoperative complications. Conclusion: This case presents the usefulness of a MR system offering organ visualization as hologram during surgery.
Keywords: rectal cancer, robotic surgery, mixed reality, holograms

Colorectal cancer is one of the most common cancers in the world (1), and surgical resection is commonly performed as curative treatment. Some randomized clinical trials have confirmed that laparoscopic surgery resulted in equivalent long-term oncologic outcomes and improved short-term results compared with traditional open surgery (2-5). Further technological advances have led to the development of robotic surgery. Robotic surgery has many technical advantages such as superior stability, curved flexible instruments, increased dexterity, fixed stable traction, and improved accuracy (6-8). Previous studies have reported that robotic surgery for colorectal cancer is associated with a shorter length of hospital stay, lower conversion rate, and lower overall complication rate than laparoscopic surgery (9-11). Surgical methods have gradually transitioned from traditional laparotomy to laparoscopic surgery and robotic surgery. Although surgical equipment has advanced, especially in the case of rectal cancer, high skill is still required to perform the surgery while correctly understanding the pelvic anatomy. Therefore, supporting tools are useful to improve the surgical efficiency. Recently, augmented and mixed reality technology has been applied to surgery (12-14). Mixed reality (MR) enables the simultaneous experience of the real world and virtual reality by precisely superimposing images in coordinate spaces. The application of mixed reality to surgery, specifically the use of head-mounted display (HMD), allows surgeons to use intraoperative mixed reality technology to visualize 3D computer graphics models (holograms) of each patient during operation (14).

Here, we report a case treated with robotic surgery performed with 3D images using HoloLens2 (Microsoft, Microsoft Corporation, Redmond, WA) as HMD during the operation. The holograms were created using preoperative positron emission tomography-computed tomography (PET-CT).

Case Report

The patient was a 41-year-old woman with a history of abdominal surgery. She was not prescribed any medicine. The patient’s family medical history was unremarkable. The patient visited a nearby physician for positive fecal occult blood, and colonoscopy was performed. She was diagnosed with rectal cancer (cT1b, cN0, cM0, stage I) and was referred to our hospital for curative treatment. Her height was 160 cm, weight was 52 kg, BMI was 20.3. Her preoperative laboratory data were as follows: leukocytes 4,770 /dl, hemoglobin 12.5 g/dl, platelets 20.1×104/μl, AST 14 U/l, ALT 8 U/l, ALP 43 IU/l, BUN 16 mg/dl, creatinine 0.63 mg/dl, CRP 0.04 mg/dl, electrolytes were within normal limits, CA19-9 9.5 U/ml, and CEA 2 ng/ml. A PET-CT and magnetic resonance imaging (MRI) revealed the localization of cancer. Robot-assisted anterior resection was performed. The operator used HoloLens2 and performed the surgery while visualizing 3D images of pelvic anatomy with the location of the rectal cancer as hologram (Figure 1, Figure 2, and Figure 3). The operation time was 261 minutes, and blood loss was 30 ml. The patient’s postoperative course was uneventful, and she was discharged 11 days after surgery, and is still relapse-free.


This case showed the utility and safety of intraoperative hologram support for rectal cancer as a previous report (15). The pelvic cavity is surrounded by blood vessels and nerves, which should not be injured, so it is important to develop support tools for improving surgical safety. The hologram image can support and provide surgeons with a better understanding of the individual patient’s pelvic anatomy. The information can’t be obtained from the CT images or operation field on a planar monitor during operation. Furthermore, the 3D model can be magnified in the hologram view, and the surgeon can move to observe the hologram from various angles. The hologram was created using preoperative PET-CT, and reflected the anatomy. Thus, it is considered that intraoperative simulations can improve the anatomical understanding compared with preoperative simulations. And if all surgeons of the surgery team wear the HoloLens2 to share the same hologram, their discussion and communication will be more active and improve the understanding of the positional relationship of the organs of the patient.


In conclusion, this study suggests that the use of intraoperative hologram support can improve the understanding of the individual’s anatomy and surgical safety. It also suggests that intraoperative holograms could be useful surgical tools to educate next-generation surgeons. Further study is needed to accumulate more cases in the future.

Conflicts of Interest

All the Authors declare no conflicts of interest in relation to this study.

Authors’ Contributions

The first draft of the manuscript was written by R.H. S.F. and N.M. commented on previous versions of the manuscript and edited. Y.D. and H.E. supervised this report. Y.S., M.T., T.H., A.H., T.O., M.U., and H.Y. discussed the results of this manuscript. All Authors read and approved the final manuscript.


1 Ferlay J Soerjomataram I Dikshit R Eser S Mathers C Rebelo M Parkin DM Forman D & Bray F Cancer incidence and mortality worldwide: Sources, methods and major patterns in GLOBOCAN 2012. Int J Cancer. 136(5) E359 - E386 2015. DOI: 10.1002/ijc.29210
2 Jayne DG Guillou PJ Thorpe H Quirke P Copeland J Smith AM Heath RM & Brown JM Randomized trial of laparoscopic-assisted resection of colorectal carcinoma: 3-year results of the UK MRC CLASICC trial group. J Clin Oncol. 25(21) 3061 - 3068 2007. DOI: 10.1200/JCO.2006.09.7758
3 Veldkamp R Kuhry E Hop WC Jeekel J Kazemier G Bonjer HJ Haglid E Pahlman L Cuesta MA Msika S Morino M Lacy AM & COlon cancer Laparoscopic or Open Resection Study Group (COLOR) Laparoscopic surgery versus open surgery for colon cancer: short-term outcomes of a randomised trial. Lancet Oncol. 6 477 - 484 2005. DOI: 10.1016/S1470-2045(05)70221-7
4 Colon Cancer Laparoscopic or Open Resection Study Group Buunen M Veldkamp R Hop WC Kuhry E Jeekel J Haglind E Pahlman L Cuesta MA Msika S Morino M Lacy A & Bonjer HJ Survival after laparoscopic surgery versus open surgery for colon cancer: long-term outcome of a randomised clinical trial. Lancet Oncol. 10 44 - 52 2009. DOI: 10.1016/S1470-2045(08)70310-3
5 Clinical Outcomes of Surgical Therapy Study Group Nelson H Sargent DJ Wieand HS Fleshman J Anvari M Stryker SJ Beart RW Jr Hellinger M Flanagan R Jr Peters W & Ota D A comparison of laparoscopically assisted and open colectomy for colon cancer. N Engl J Med. 350 2050 - 2059 2004. DOI: 10.1056/NEJMoa032651
6 Aly EH Robotic colorectal surgery: summary of the current evidence. Int J Colorectal Dis. 29(1) 1 - 8 2014. DOI: 10.1007/s00384-013-1764-z
7 Kim J Baek S Kang D Roh Y Lee JW Kwak H Kwak JM & Kim S Robotic resection is a good prognostic factor in rectal cancer compared with laparoscopic resection: long-term survival analysis using propensity score matching. Dis Colon Rectum. 60(3) 266 - 273 2017. DOI: 10.1097/DCR.0000000000000770
8 Kuo L Ngu JC Tong Y & Chen C Combined robotic transanal total mesorectal excision (R-taTME) and single-site plus one-port (R-SSPO) technique for ultra-low rectal surgery—initial experience with a new operation approach. Int J Colorectal Dis. 32(2) 249 - 254 2017. DOI: 10.1007/s00384-016-2686-3
9 He C Wang X Cao G Mao W & Lao W Robot-assisted versus laparoscopic surgery for rectal cancer: A systematic review and meta-analysis. J Can Res Ther. 16(5) 979 2020. DOI: 10.4103/jcrt.JCRT_533_18
10 Zhang Y Feng H Wang S Gu Y Shi Y Song Z Deng Y Ji X Cheng X Zhang T & Zhao R Short- and long-term outcomes of robotic- versus laparoscopic-assisted right hemicolectomy: A propensity score-matched retrospective cohort study. Int J Surg. 105 106855 2022. DOI: 10.1016/j.ijsu.2022.106855
11 Feng Q Yuan W Li T Tang B Jia B Zhou Y Zhang W Zhao R Zhang C Cheng L Zhang X Liang F He G Wei Y Xu J & REAL Study Group Robotic versus laparoscopic surgery for middle and low rectal cancer (REAL): short-term outcomes of a multicentre randomised controlled trial. Lancet Gastroenterol Hepatol. 7 991 - 1004 2022. DOI: 10.1016/S2468-1253(22)00248-5
12 Müller F Roner S Liebmann F Spirig JM Fürnstahl P & Farshad M Augmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3D imaging. Spine J. 20(4) 621 - 628 2020. DOI: 10.1016/j.spinee.2019.10.012
13 Neves CA Vaisbuch Y Leuze C Mcnab JA Daniel B Blevins NH & Hwang PH Application of holographic augmented reality for external approaches to the frontal sinus. Int Forum Allergy Rhinol. 10(7) 920 - 925 2020. DOI: 10.1002/alr.22546
14 Saito Y Sugimoto M Imura S Morine Y Ikemoto T Iwahashi S Yamada S & Shimada M Intraoperative 3D hologram support with mixed reality techniques in liver surgery. Ann Surg. 271(1) e4 - e7 2020. DOI: 10.1097/SLA.0000000000003552
15 Tarutani K Takaki H Igeta M Fujiwara M Okamura A Horio F Toudou Y Nakajima S Kagawa K Tanooka M & Yamakado K Development and accuracy evaluation of augmented reality-based patient positioning system in radiotherapy: a phantom study. In Vivo. 35(4) 2081 - 2087 2021. DOI: 10.21873/invivo.12477