Exploring Diffusion Models and Self-Supervised ViT's in Generating and Indexing Synthetic Data to Enhance Polyp Segmentation

Abstract

Polyps in the gastrointestinal (GI) tract is a known precursor to colorectal cancer (CRC), making detection and removal essential for cancer prevention. Despite the effectiveness of screening procedures like colonoscopies, high miss rates of up to 28 % calls for computer aided detection systems (CAD) based on deep learning to increase detection rates. Scarcity of labeled medical data as a result of strict privacy laws, however, serves as a major bottleneck in the development of detection and segmentation models. This thesis explores the efficacy of using diffusion models and self-supervised vision transformers to subsequently generate polyp images and complex embed- dings, in an effort to enhance polyp segmentation performance. By performing in- dexing based on embeddings for both real and synthetic data, we create synthetically infused datasets for training segmentation models. Generated data is split into cat- egories based on similarity and variation within the embedded representations com- pared to real samples. We showcase the ability of conditional latent diffusion models to generate highly realistic synthetic polyps and GI tract images. We further display the applic- ations of using self-supervised vision transformers to create rich representations of images capturing deep features and semantics within polyps and the GI tract, en- abling us to create enhanced datasets containing infused generated images based on characteristics of their embeddings. From experiments, segmentation models trained on synthetically enriched datasets show increased performance in-domain, and sig- nificant increases out-of-domain, compared to using only real data. Additionally, we identify that generated polyp images with embeddings displaying similar character- isitc to real samples improve segmentation performance the most. As a conclusion, the integration deep learning concepts such as generative models and self-supervised vision transformers have the capacity to significantly improve detection and segmentation performance of polyps in the GI tract. i