Deep learning in volume rendering

Abstract

A variety of scientific fields commonly acquire volumetric data, for example by using computed tomography (CT) or magnetic resonance imaging (MRI) in medicine. Such volumetric data is often complex and requires visualization to gain understanding. However, rendering volumetric data entails several challenges, such as classifying the data to reveal the structures of interest. Meaningful classification of 3D data is difficult to implement in 2D graphical user interfaces. Furthermore, rendering of volumetric data is generally compute intensive, especially when considering volumetric shading. Lastly, volume rendering needs to be interactive and achieve reasonable frame rates in order to support fully exploring the 3D data from different views, while adapting the classification. To combat these challenges, this work explores how volume rendering and its individual aspects can be assisted by means of deep neural networks. Deep neural networks have recently proven very competent in many disciplines, like natural language processing, vision, but also computer graphics. They excel in the approximation of complex functions and can learn relevant features and representations when trained with sufficient data. In the scope of this dissertation, we show how these capabilities can be used throughout the volume rendering pipeline. This pipeline consists of the classification of structures of interest, shading of those structures and the composition of the light transmitted through the volume. In the classification step, we leverage the strong representations learned by self-supervised neural networks to enable an interactive click-to-select workflow that segments structures annotated by users within slice views. For shading, we propose a volume-to-volume network to predict volumetric ambient occlusion that respects how the volume is classified. Lastly, we employ a deep neural network to invert the composition step, separating different structures in an already composited semi-transparent volume rendered image into a modifiable layered representation.

In reference to IEEE copyrighted material which is used with permission in this thesis, the IEEE does not endorse any of [university/educational entity's name goes here]'s products or services. Internal or personal use of this material is permitted. If interested in reprinting/republishing IEEE copyrighted material for advertising or promotional purposes or for creating new collective works for resale or redistribution, please go to http://www.ieee.org/publications_standards/publications/rights/rights_link.html to learn how to obtain a License from RightsLink.