Performance is not Boolean: Supporting Scalar Configuration Variables in NFP Models

Abstract

Non-functional properties (NFPs) such as memory requirements, timing, or energy consumption are important characteristics of embedded software systems and software product lines (SPLs) in general. Both during system design and at runtime, the goal is to optimize resource utilization (and, thus, NFPs) by appropriate system configuration or orchestration. NFP models, learned from benchmarks of various SPL configurations, allow for the prediction of these properties, thus enabling NFP-aware software configuration and runtime decisions. However, many existing approaches for automated learning of NFP models limit their scope to boolean variables. We argue that this is no longer sufficient: NFP models must accommodate scalar variables to achieve suitable accuracy when faced with today’s highly configurable software systems and variable workloads. To this end, we evaluate four regression tree-based NFP modeling approaches on eight use cases, and examine model complexity and model accuracy. We find that models with support for scalar variables achieve up to three times lower mean model error when predicting configurations that were not part of the training set. At the same time, the complexity of scalar and boolean-only models is nearly the same; only benchmarking becomes more time-intensive due to the need to explore scalar variables. We conclude that scalar-enabled models provide increased accuracy almost free of charge, and recommend using them when generating NFP models for embedded systems and workloads with scalar configuration variables.