Project

Back to overview

ASysT: Automatic System Testing

English title ASysT: Automatic System Testing
Applicant Pezzè Mauro
Number 162409
Funding scheme Project funding (Div. I-III)
Research institution Software Institute Facoltà di scienze informa Universit`a della Svizzera italiana
Institution of higher education Università della Svizzera italiana - USI
Main discipline Information Technology
Start/End 01.10.2015 - 30.06.2018
Approved amount 302'212.00
Show all

Keywords (7)

Software testing; Software reliability; Automatic test cases generation; Automatic test case execution; GIU testing; Q-Learning; Semantic Web

Lay Summary (Italian)

Lead
Moltissime attività personali, sociali e commerciali si affidano ad applicazioni mobili, servizi web e sistemi commerciali che sono diventati essenziali nella nostra vita quotidiana. Il test di queste applicazioni è un’attività manuale noiosa, costosa e spesso trascurata nella corsa contro tempi di consegna che devono essere sempre più rapidi e contro costi che si vogliono sempre più bassi. Questo fa sì che molte applicazioni interattive presentino difetti che possono portare a risultati scorretti e a problemi latenti con impatti anche gravi su affidabilità, sicurezza e confidenzialità di molte applicazioni interattive.In questo progetto svilupperemo un approccio completamente automatico per la generazione e l’esecuzione di insiemi di test in grado di sollecitare tutti i comportamenti delle applicazioni e mettere in evidenza eventuali difetti che potranno quindi essere corretti prima di portare a malfunzionamenti sul campo.
Lay summary

Moltissime attività personali, sociali e commerciali si affidano ad applicazioni mobili, servizi web e sistemi commerciali che sono diventati essenziali nella nostra vita quotidiana.  Tutte queste applicazioni, che indichiamo con il termine di applicazioni interattive, richiedono input complessi da parte dell’utente in forma di sequenze di azioni su menu, pulsanti e altro e in forma di moduli che richiedono informazioni semanticamente rilevanti.

Il test di queste applicazioni è un’attività manuale noiosa e costosa che viene spesso trascurata nella corsa contro tempi di consegna che devono essere sempre più rapidi e costi che si vogliono sempre più bassi. Questo fa sì che molte applicazioni interattive presentino difetti che possono portare a risultati scorretti e a problemi latenti con impatti anche gravi su affidabilità, sicurezza e confidenzialità di molte applicazioni interattive.

In questo progetto svilupperemo un approccio completamente automatico per la generazione e l’esecuzione di insiemi di test in grado di sollecitare tutti i comportamenti delle applicazioni e mettere in evidenza eventuali difetti che potranno quindi essere corretti prima di portare a malfunzionamenti sul campo.

Nel progetto svilupperemo tecniche di analisi delle interfacce per identificare me modalità di interazione con l’utente e utilizzeremo tecniche di apprendimento automatico e il web semantico per produrre test significativi in modo automatico.

L’ambiente di test automatico prodotto nel progetto potrà essere integrato in ambienti di sviluppo e di produzione per permettere attività di test continuo durante lo sviluppo e di certificazione prima della distribuzione dell’applicazione finale

Direct link to Lay Summary Last update: 25.09.2015

Responsible applicant and co-applicants

Employees

Publications

Publication
Augusto: Exploiting Popular Functionalities for the Generation of Semantic GUI Tests with Oracles
Mariani Leonardo, Pezzè Mauro, Zuddas Daniele (2018), Augusto: Exploiting Popular Functionalities for the Generation of Semantic GUI Tests with Oracles, in Proceedings of the International Conference on Software Engineering, IEEE, New York.
Automatic GUI Testing of Desktop Applications: an Empirical Assessment of the State of the Art
Pezzè Mauro (2018), Automatic GUI Testing of Desktop Applications: an Empirical Assessment of the State of the Art, in INTUITESTBEDS Proceedings of the 4th International Workshop on User Interface Test Automation, AmserdamACM, New York.
Effectiveness and challenges in generating concurrent tests for thread-safe classes
Terragni Valerio, Pezze' Mauro (2018), Effectiveness and challenges in generating concurrent tests for thread-safe classes, in 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE), ACM, New York.
Reproducing concurrency failures from crash stacks
Francesco Bianchi, Pezze' Mauro (2017), Reproducing concurrency failures from crash stacks, in Proceedings of the 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE, PaderbornACM, New York.
Semantic Testing of Interactive Applications
Zuddas Daniele (2016), Semantic Testing of Interactive Applications, in Companion proceedings of the International Conference on Software Testing, Verification and Validati, 391-392, ACM, New York391-392.
Recent Advances in Automatic Black-Box Testing
Mariani Leonardo, Pezzè Mauro, Zuddas Daniele (2015), Recent Advances in Automatic Black-Box Testing, Elsevier, Berlin, 157-193.

Collaboration

Group / person Country
Types of collaboration
Università di Milano Bicocca Italy (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication

Scientific events

Active participation

Title Type of contribution Title of article or contribution Date Place Persons involved
ECOOP/ISSTA 4th International Workshop on User Interface Test Automation Talk given at a conference Automatic GUI Testing of Desktop Applications: an Empirical Assessment of the State of the Art 15.07.2018 Amsterdam, Sweden Pezzè Mauro; Zuddas Daniele;
ICSE International Conference on Software Engineering Talk given at a conference Exploiting Popular Functionalities for the Generation of Semantic GUI Tests 27.05.2018 Gothenburg, Sweden Zuddas Daniele; Pezzè Mauro;
International Conference on Software Engineering (ICSE) Talk given at a conference Heuristically matching solution spaces of arithmetic formulas to efficiently reuse solutions 22.05.2017 Buenos Aires, Argentina Pezzè Mauro;
ACM Foundations of Software Engineering (FSE) Talk given at a conference JBSE: a symbolic executor for Java programs with complex heap inputs 14.11.2016 Seattle, United States of America Pezzè Mauro;
ACM International Symposium on Software Testing and Analysis (ISSTA) Talk given at a conference Automatic generation of oracles for exceptional behaviors 18.07.2016 Saarbrücken, Germany Pezzè Mauro;
IEEE International Conference on Software Testing - Doctoral Symposium Talk given at a conference Semantic Testing of Interactive Applications 10.04.2016 Chicago, United States of America Zuddas Daniele; Pezzè Mauro;


Knowledge transfer events

Active participation

Title Type of contribution Date Place Persons involved
Oracle Cloud Day Talk 09.11.2016 Brugg, Switzerland Pezzè Mauro;
IBM Interconnect Talk 22.02.2016 Las Vegas, United States of America Pezzè Mauro;


Associated projects

Number Title Start Funding scheme
178742 ASTERIx: Automatic System TEsting of inteRactive software applIcations 01.04.2018 Project funding (Div. I-III)

Abstract

In this project we will develop a new approach to automatically test the many applications that we use everyday though the Web and our mobile devices. In detail, we will develop a method to automatically generate system test cases, targeting the problem of testing the common class of software systems that elaborate complex input data characterised by different kinds of mutually dependent values, and we will define approaches and techniques to automatically generate complex sequences of mutually dependent and realistic data that will extensively sample the behavior of the system under test. System testing is inherently different from unit and integration testing due to the nature, dimension and articulation of the execution space. The nature of the execution space is different because system testing targets the behavior of the system as a whole, for example, sorting a list of contacts, while unit and integration testing targets the behavior of units and their interactions independently for the overall execution context, for example, sorting strings. Thus, test case generation techniques that work at the system level shall generate semantically relevant input data, like realistic contacts and not just strings. The dimension of the execution space of a system may include a big variety of different data, while the execution space of single units and subsystems focuses on the much smaller subset of data relevant for the unit or the subsystem. Systematically exploring the execution space at the system level becomes prohibitive, thus defining effective heuristics for sampling the input space is extremely important to automate system testing. The articulation of the execution space at system level differs from unit and integration level: while units and subsystems can be tested in isolation, system testing requires testing a system while interacting with the environment. Thus system test cases must represent both meaningful sequences of operations and input data that represent appropriate interactions with the environment, that is, interaction sequences that lead to successful executions and data that refer to realistic and semantically coherent inputs. So far, the research on automatic test case generation techniques has focused on unit and integration testing, paying less attention to system testing. The few approaches to system testing are based on models, learning techniques, search-based algorithms and reuse. These strategies do not tackle the problem of generating legal operation sequences and realistic and coherent test inputs, and can hardly cope with applications that extensively exploit the semantic of the input data. In this project we will define a new approach to automatically generate semantically coherent and effective system test suites for applications that require complex sequences of mutually dependent data. The research will be grounded on our recent work and will (i) investigate the possibility of using both knowledge from a core set of correct executions and semantic information from system specifications to extend the ability of machine learning techniques, and in particular Q-learning, to deal with complex sequences of operation, (ii) study the integration of different knowledge bases, starting with the Web of data, to address specific application domains, (iii) define techniques that support an efficient interplay between the different approaches by exploring complementarities and similarities, (iv) define techniques to deal with semantic correlations between data used in different operations, (v) define approaches to efficiently deal with negative cases, (vi) complement the approaches with techniques to automatically generate partial oracles, and (vii) experimentally evaluate the approach with suitable prototypes referring to applications from different domains and based on different technologies.
-