Shifting Quality Assurance of Machine Learning Algorithms to Live Systems

Abstract

A fundamental weakness of existing solutions to assess the quality of machine learning algorithms is the assumption that test environments sufficiently mimic the later application. Given the data dependent behavior of these algorithms, only limited reasoning about their later performance is possible. Thus, meaningful quality assurance is not possible with test environments. A shift from the traditional testing environment to the live system is needed. Thus, costly test environments are replaced with available live systems that constantly execute the algorithm.