Publication

Back to overview

A Systematic Literature Review of Software Visualization Evaluation

Type of publication Peer-reviewed
Publikationsform Original article (peer-reviewed)
Author Merino Leonel, Ghafari Mohammad, Anslow Craig, Nierstrasz Oscar,
Project Agile Software Analysis
Show all

Original article (peer-reviewed)

Journal Journal of Systems and Software
Volume (Issue) 144
Page(s) 165 - 180
Title of proceedings Journal of Systems and Software
DOI 10.1016/j.jss.2018.06.027

Open Access

URL http://scg.unibe.ch/archive/papers/Meri18a.pdf
Type of Open Access Website

Abstract

Abstract Context: Software visualizations can help developers to analyze multiple aspects of complex software systems, but their effectiveness is often uncertain due to the lack of evaluation guidelines. Objective: We identify common problems in the evaluation of software visualizations with the goal of formulating guidelines to improve future evaluations. Method: We review the complete literature body of 387 full papers published in the SOFTVIS/VISSOFT conferences, and study 181 of those from which we could extract evaluation strategies, data collection methods, and other aspects of the evaluation. Results: Of the proposed software visualization approaches, 62% lack a strong evaluation. We argue that an effective software visualization should not only boost time and correctness but also recollection, usability, engagement, and other emotions. Conclusion: We call on researchers proposing new software visualizations to provide evidence of their effectiveness by conducting thorough (i) case studies for approaches that must be studied in situ, and when variables can be controlled, (ii) experiments with randomly selected participants of the target audience and real-world open source software systems to promote reproducibility and replicability. We present guidelines to increase the evidence of the effectiveness of software visualization approaches, thus improving their adoption rate.
-