Domain-specific foundation models for science applications: Self-supervised learning with SAR and DXA

Abstract

This dissertation explores the use of self-supervised pre-training in non-natural image domains andprovides three main contributions to the literature: 1) it adapts current self-supervised learning frameworks to pre-train a model specific to synthetic aperture radar Wave mode imagery; 2) it adapts current self-supervised learning frameworks to pre-train a model specific to dual-energy x-ray absorptiometry; 3) it analyzes embedding characteristics of both models to identify representation quality metrics effective beyond natural image applications. The immediate goal of this work is to provide embedding models that generalize effectively to a range of downstream tasks in their respective domains. These models serve as highly specific foundation models — they generalize well to in-domain tasks, they are robust to training settings and hyperparameter choices, and they are extremely labeled-data-efficient. Training models with self-supervised methods that are tuned to the characteristics of the data domain is important because most self-supervised frameworks are highly tuned for optimal perfor- mance on natural images. By adapting these frameworks to respect domain-specific characteristics of the data or simply removing natural-image-focused biases, downstream task performance and generalizability can be improved. The secondary goal is to add to the body of literature exploring representation characteristics, searching for embedding space qualities that indicate a well-performing model without access to labeled data for direct evaluation. This addresses a crucial bottleneck for similar domain-specific pre-training efforts where architecture search, hyperparameter tuning, and comparison between self-supervised methods are all hindered by the need to train each candidate model to completion and evaluate performance on specific downstream tasks. By training novel embedding models for two separate vision domains and extensively analyzing intermediate representations of successful and unsuccessful models, this study seeks to establish a foundation for future research attempting similar pre-training efforts for other computer vision domains beyond natural images.