Gemini connector

Abstract

Spectacular advances have been made in the field of machine vision over the past decade. While this discipline is traditionally driven by geometric models, neural networks have proven to be superior in some applications and have significantly expanded the limits of what is possible. At the same time, conventional graphic models describe the relationship between images and the associated scene with textures and light in a physically realistic manner and are an important part of photogrammetry. Differential renderers combine these approaches by enabling gradient-based optimization in fixed structures of a graphics pipeline and thus adapt the learning process of neural networks. This fusion of formalized knowledge and machine learning motivates the idea of a modular differentiable renderer in which physical and statistical models can be recombined depending on the use case. We therefore present Gemini Connector: an initiative for the modular development and combination of differentiable physical models and neural networks. We examine opportunities and problems and motivate the idea with the extension of a differentiable rendering pipeline to include models of underwater optics for the analysis of deep sea images. Finally, we discuss use cases, especially within the Cross-Domain Fusion initiative.