The impact of test ownership and team structure on the reliability, effectivenessof quality test runs

Abstract

This submission presents work submitted and accepted at the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement in 2014 [Hn2014]. Software testing is a crucial step in most software development processes. Testing software is a key component to manage and assess the risk of shipping quality products to customers. But testing is also an expensive process and changes to the system need to be tested thoroughly which may take time. Thus, the quality of a software product depends on the quality of its underlying testing process and on the effectiveness and reliability of individual test cases. In this paper, we investigate the impact of the organizational structure of test owners on the reliability and effectiveness of the corresponding test cases. Prior empirical research on organizational structure has focused only on developer activity. We expand the scope of empirical knowledge by assessing the impact of organizational structure on testing activities. We performed an empirical study on the Windows build verification test suites (BVT) and relate efficiency and reliability measures of each test run to the complexity and size of the organizational sub-structure that enclose all owners of test cases executed. Our results show, that organizational structure impacts both test efficiency and test execution reliability. We are also able to predict effectiveness and reliability with fairly high precision and recall values.