Fooling the masses with performance results: Old classics and some new ideas

Abstract

G. Wellein and G. Hager Department for Computer Science and Erlangen Regional Computing Center Friedrich-Alexander-Universität Erlangen-Nürnberg In 1991, David H. Bailey published his insightful 'Twelve Ways to Fool the Masses When Giving Performance Results on Parallel Computers.' In that humorous article, Bailey pinpointed typical 'evade and disguise' techniques for presenting mediocre performance results in the best possible light. At that time, the supercomputing landscape was governed by the 'chicken vs. oxen' debate: Could strong vector CPUs survive against the new massively parallel systems? In the past two decades, hybrid, hierarchical systems, multi-core processors, accelerator technology, and the dominating presence of commodity hardware have reshaped the landscape of High Performance Computing. It's also not so much oxen vs. chickens anymore; billions of ants have entered the battlefield. This talk gives an update of the 'Twelve Ways.' Old classics are presented alongside new 'stunts' that reflect today's technological boundary conditions. DISCLAIMER: Although these musings are certainly inspired by experience with many publications and talks in HPC, I wish to point out that (i) no offense is intended, (ii) I am not immune to the inherent temptations myself and (iii) this all still just meant to be fun.