Project

Back to overview

Test Code Quality: A New Dimension of Test Code Effectiveness

Applicant Palomba Fabio
Number 186090
Funding scheme Ambizione
Research institution Institut für Informatik Universität Zürich
Institution of higher education University of Zurich - ZH
Main discipline Information Technology
Start/End 01.09.2019 - 31.08.2023
Approved amount 507'836.00
Show all

Keywords (4)

Software Quality; Software Testing; Empirical Software Engineering; Machine Learning

Lay Summary (Italian)

Lead
Fabio Palomba
Lay summary

Nella nostra società, il software è sempre più al centro di qualunque attività quotidiana: dalle transazioni finanziarie ai trattamenti sanitari, e altro ancora, facciamo affidamento sui sistemi software. Sfortunatamente, in un tale contesto, l'emergere di gravi difetti software che a volte portano a conseguenze sociali drammatiche è la regola piuttosto che l'eccezione, come testimoniamo troppo spesso le notizie sui giornali. Pertanto, la creazione di meccanismi che possano garantire il corretto funzionamento del software ed evitare l'introduzione di difetti è della massima urgenza. Il progetto TEst coDe (TED) mira a definire e sviluppare nuove metodologie e tecniche automatizzate che consentono agli sviluppatori software di (1) migliorare il testing di sistemi software, così da poter meglio controllare la presenza di difetti nel codice sorgente che producono e (2) monitorare l'adeguatezza dei meccanismi che utilizzano per verificare l'affidabilità del software. Inoltre, il progetto valuterà la fattibilità del cosidetto crowd-testing, ovvero l'integrazione del ragionamento umano nel processo di valutazione dell'affidabilità del software. I risultati di questo progetto saranno una comprensione scientifica più profonda del testing e dell’affidabilità di sistemi software e una serie di strumenti automatici che consentiranno agli sviluppatori software di avere una migliore qualità del software.

Direct link to Lay Summary Last update: 23.08.2019

Responsible applicant and co-applicants

Employees

Publications

Publication
The Relation of Test-Related Factors to Software Quality: A Case Study on Apache Systems
Pecorelli Fabiano, Palomba Fabio, De Lucia Andrea (2021), The Relation of Test-Related Factors to Software Quality: A Case Study on Apache Systems, in Empirical Software Engineering, 26(2), 18-18.
Within-Project Defect Prediction of Infrastructure-as-Code Using Product and Process Metrics
Dallapalma Stefano, Di Nucci Dario, Palomba Fabio, Tamburri Damian Andrew (2021), Within-Project Defect Prediction of Infrastructure-as-Code Using Product and Process Metrics, in IEEE Transactions on Software Engineering, 1-1.
tsDetect: an open source test smells detection tool
Peruma Anthony, Almalki Khalid, Newman Christian D., Mkaouer Mohamed Wiem, Ouni Ali, Palomba Fabio (2020), tsDetect: an open source test smells detection tool, in ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundatio, Virtual Event USAACM, ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundatio.
VITRuMA Plug-In for the Visualization of Test-Related Metrics
Pecorelli Fabiano, Di Lillo Gianluca, Palomba Fabio, De Lucia Andrea (2020), VITRuMA Plug-In for the Visualization of Test-Related Metrics, in AVI '20: International Conference on Advanced Visual Interfaces, Salerno ItalyAVI, AVI '20: International Conference on Advanced Visual Interfaces.
Pizza versus Pinsa: On the Perception and Measurability of Unit Test Code Quality
Grano Giovanni, De Iaco Cristian, Palomba Fabio, Gall Harald C. (2020), Pizza versus Pinsa: On the Perception and Measurability of Unit Test Code Quality, in 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), Adelaide, AustraliaIEEE, 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME).
Just-In-Time Test Smell Detection and RefactoringThe DARTS Project
Lambiase Stefano, Cupito Andrea, Pecorelli Fabiano, De Lucia Andrea, Palomba Fabio (2020), Just-In-Time Test Smell Detection and RefactoringThe DARTS Project, in ICPC '20: 28th International Conference on Program Comprehension, Seoul Republic of KoreaACM, ICPC '20: 28th International Conference on Program Comprehension.
Testing of Mobile Applications in the WildA Large-Scale Empirical Study on Android Apps
Pecorelli Fabiano, Catolino Gemma, Ferrucci Filomena, De Lucia Andrea, Palomba Fabio (2020), Testing of Mobile Applications in the WildA Large-Scale Empirical Study on Android Apps, in ICPC '20: 28th International Conference on Program Comprehension, Seoul Republic of KoreaACM, ICPC '20: 28th International Conference on Program Comprehension.
On the performance of method-level bug prediction: A negative result
Pascarella Luca, Palomba Fabio, Bacchelli Alberto (2020), On the performance of method-level bug prediction: A negative result, in Journal of Systems and Software, 161, 110493-110493.
Why Don't Developers Detect Improper Input Validation?'; DROP TABLE Papers; --
BrazLarissa, FregnanEnrico, CalikliGul, BacchelliAlberto, Why Don't Developers Detect Improper Input Validation?'; DROP TABLE Papers; --, in 43rd International Conference on Software Engineering (ICSE 2021), IEEE, 43rd International Conference on Software Engineering (ICSE 2021).

Collaboration

Group / person Country
Types of collaboration
Software Engineering Maintenance and Evolution Research Unit (SEMERU) United States of America (North America)
- Publication
Software Evolution and Architecture Lab (SEAL) Switzerland (Europe)
- in-depth/constructive exchanges on approaches, methods or results
IMDEA Software Institute Spain (Europe)
- Publication
Software Engineering and Technology Group (SET) Netherlands (Europe)
- Publication
Software Engineering Research Group (SERG) Netherlands (Europe)
- Publication

Abstract

Our society runs on software: Ranging from financial transactions to health treatments, and more, we rely on software systems. Unfortunately, in such a context the emergence of serious faults that sometimes lead to dramatical societal consequences is the rule rather than the exception, as we witness too often in the news. Therefore, the creation of mechanisms to ensure the correct functioning of software and avoid the introduction of faults is of the greatest urgency.Software testing is a crucial activity that developers perform to produce high-quality and reliable software. However, determining the effectiveness of software tests is an open research and practical challenge. To assess test effectiveness, many organizations rely on metrics like branch or mutation coverage; unfortunately, the academic community pointed out that those metrics are not optimal as they (i) cannot provide a comprehensive view of test effectiveness and (ii) are computationally too expensive to be computed within modern development contexts such as Continuous Integration. With the project presented in this proposal, I aim at facing these fundamental challenges, thus bringing testing research to the next level. First, I propose to reformulate the concept of test code effectiveness considering multiple aspects at different levels of granularity and build a comprehensive theory of the factors impacting test code effectiveness, with the goal of automating the identification of test instabilities. Among such factors, I conjecture that test code quality is a fundamental dimension of test code effectiveness, which has been poorly considered in the past: For instance, the research my co-authors and I conducted has provided initial compelling evidence that the design quality of the test cases relates to serious problems, such as test flakiness and limited ability to find faults in production code. Second, I aim at facing major challenges related to developers' awareness and testing adoption. I propose the definition of a new generation of test code checkers that may precisely pinpoint the location of test instabilities, quantify their impact on test code effectiveness adopting machine learning techniques, and enable a continuous runtime feedback on the improvement of the production code due to tests. Finally, the key idea of my research proposal is to ease the adoption of testing practices by (1) defining the concept of machine learning-guided automatic test case generation and repair, where the information related to the most-impacting test instabilities coming from machine learning analyses done by the devised test code checkers are used as input of search-based algorithms that optimize the generation/repair of individual tests based on the identified properties, thus making the generation dynamic in nature and dependent on the need of a test; and (2) exploiting crowdsourcing mechanisms with the aim of integrating human reasoning into test generation, thus complementing the capabilities of automatic test generators.The expected long-term benefits of this research are a deeper scientific understanding of testing, improved test effectiveness, and better quality software. My strong track record in software testing, test code quality, software analytics, and empirical software engineering, together with the envisioned collaborations within the Department of Informatics and the University of Zurich, form a unique basis that enables me to perform and lead this project. My unique international collaborations with top scientists in the field enable high-quality scientific results and a rapid industry uptake of the results.
-