Publications

2024

1. VaryMinions: Leveraging RNNs to Identify Variants in Variability-intensive Systems’ Logs

   Sophie Fortz , Paul Temple , Xavier Devroey , Patrick Heymans , and Gilles Perrouin

   Empirical Software Engineering, Mar 2024

   Abs Website

   From business processes to course management, variability-intensive software systems (VIS) are now ubiquitous. One can configure these systems’behaviour by activating options, e.g., to derive variants handling building permits across municipalities or implementing different functionalities (quizzes, forums) for a given course. These customisation facilities allow VIS to support distinct relevant customer requirements while taking advantage of reuse for common parts. Customisation thus allows realising both scope and scale economies. Behavioural differences amongst variants manifest themselves in event logs. To re-engineer this kind of system, one must know which variant(s) have produced which behaviour. Since variant information is barely present in logs, this paper supports this task by employing machine learning techniques to classify behaviours (event sequences) among variants. Specifically, we train Long Short Term Memory (LSTMs) and Gated Recurrent Units (GRUs) recurrent neural networks to relate event sequences with the variants they belong to on six different datasets issued from the configurable process and VIS domains. After having evaluated 20 different architectures of LSTM/GRU, our results demonstrate that it is possible to effectively learn the trace-to-variant mapping with high accuracy (at least 80% and up to 99%) and at scale, i.e., identifying 50 variants using 5000+ traces for each variant.

2. Exploring the Computational Complexity of SAT Counting and Uniform Sampling with Phase Transitions

   Olivier Zeyen , Maxime Cordy , Gilles Perrouin, and Mathieu Acher

   In 46th International Conference on Software Engineering , Mar 2024

   Abs

   Uniform Random Sampling (URS) and Model Counting (#SAT) are two intrinsically linked, theoretical problems with relevant practical applications in software engineering. In particular, in configurable system engineering URS and #SAT can support study of configurations’ properties unbiasedly. Despite the community efforts to provide scalable URS and #SAT tools, solving these problems efficiently remains challenging for a large number of formulas. Contrary to the classical SAT problem, whose complexity has been an object of fundamental studies, little is known about what makes a formula hard to count or sample from. For the first time, we investigate how phase transitions can explain the practical complexity of counting and sampling. Our results, computed on 11,409 synthetic formulas and 503 real-world formulas, show that phase transitions occur in both cases, but at a different clause-to-variable ratio than for SAT tasks. We further reveal that low formula modularity is correlated with a higher URS/#SAT time. Overall, our work contributes to a principled understanding of URS and #SAT complexity.

3. FairPipes: Data Mutation Pipelines for Machine Learning Fairness

   Camille Molinier , Paul Temple , and Gilles Perrouin

   In 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024) , Mar 2024

   Abs

   Machine Learning (ML) models are ubiquitous in decision-making applications impacting citizens’ lives: credit attribution, crime recidivism, etc. In addition to seeking high performance and generalization abilities, ensuring that ML models do not discriminate against citizens regarding their age, gender, or race is essential. To this end, researchers developed various \emphfairness assessment techniques, comprising fairness metrics and mitigation approaches, notably at the model level. However, the sensitivity of ML models to fairness data perturbations has been less explored. This paper presents mutation-based pipelines to emulate fairness variations in the data once the model is deployed. FairPipes implements mutation operators that shuffle sensitive attributes, add new values, or affect their distribution. We evaluated FairPipes on seven ML models over three datasets. Our results highlight different fairness sensitivity behaviors across models, from the most sensitive perceptrons to the insensitive support vector machines. We also consider the role of model optimization in fairness performance, being variable across models. FairPipes automates fairness testing at deployment time, informing researchers and practitioners on the fairness sensitivity evolution of their ML models.

4. Preprocessing is What You Need: Understanding and Predicting the Complexity of SAT-based Uniform Random Sampling

   Olivier Zeyen , Maxime Cordy , Gilles Perrouin, and Mathieu Acher

   In 12 International Conference On Formal Methods In Software Engineering , Mar 2024

   Abs

   Despite its NP-completeness, the Boolean satisfiability problem gave birth to highly efficient tools that are able to find solutions to a Boolean formula and compute their number.Boolean formulae compactly encode huge, constrained search spaces for variability-intensive systems, e.g., the possible configurations of the Linux kernel.These search spaces are generally too big to explore exhaustively, leading most testing approaches to sample afew solutions before analysing them. A desirable property of such samples is \textituniformity: each solution should get the same selection probability.This property motivated the design of uniform random samplers, relying on SAT solvers and counters and achieving different tradeoffs between uniformity and scalability.Though we can observe their performance in practice, understanding the complexity these tools face and accurately predicting it is an under-explored problem.Indeed, structural metrics such as the number of variables and clauses involved in a formula poorly predict thesampling complexity. More elaborated ones, such as minimal independent support (MIS), are intractable to compute on large formulae.We provide an efficient parallel algorithm to compute a related metric, the \textitnumber of equivalence classes, and demonstrate that this metric is highly correlated to time and memory usage of uniform random sampling and model counting tools. We explore the role of formula preprocessing on various metrics and show its positive influence on correlations. Relying on these correlations, we train an efficient classifier (F1-score 0.97) to predict whetheruniformly sampling a given formula will exceed a specified budget. Our results allow us tocharacterise the similarities and differences between (uniform) sampling, solving and counting.

5. Time for Networks: Mutation Testing for Timed Automata Networks

   David Cortés , James Jerson Ortiz Vega , Davide Basile , Jesús Alexander Aranda Bueno , Gilles Perrouin, and Pierre-Yves Schobbens

   In 12 International Conference On Formal Methods In Software Engineering , Mar 2024

   Abs

   Mutation Testing (MT) is a technique employed to assess the effi- cacy of tests by introducing artificial faults, known as mutations, into the system. The goal is to evaluate how well the tests can detect these mutations. These artificial faults are generated using mutation operators, which produce a set of mutations derived from the original system. Mutation operators and frameworks exist for a variety of programming languages, and model-based mutation testing is gaining traction, particularly for timed safety-critical systems. This paper focuses on extending MT to Networks of Timed Automata (NTAs), an area that has not been extensively explored. We introduce mutation operators designed for NTAs specified in UPPAAL, aiming to create temporal interaction faults. We assess the effectiveness of these operators on five UPPAAL NTAs sourced from the literature, specifically examining the generation of equivalent and duplicate mutants. Our results demonstrate a varied prevalence of equivalent mutants (from 12% to 71%) while the number of duplicates is less. In all cases, timed bisimulation was able to process each mutant pair in less than one second.

6. Towards Feature-based ML-enabled Behaviour Location

   Sophie Fortz , Paul Temple , Xavier Devroey , and Gilles Perrouin

   In Proceedings of the 18th International Working Conference on Variability Modelling of Software-Intensive Systems (VaMoS 2024) , Feb 2024

   Abs

   Mapping behaviours to the features they relate to is a prerequisite for variability-intensive systems (VIS) reverse engineering. Manually providing this whole mapping is labour-intensive. In black-box scenarios, only execution traces are available (e.g., process mining). In our previous work, we successfully experimented with variant-based mapping using supervised machine learning (ML) to identify the variants responsible of the production of a given execution trace, and demonstrated that recurrent neural networks (RNNs) work well (above 80% accuracy) when trained on datasets in which we label execution traces with variants. However, this mapping (i) may not scale to large VIS because of combinatorial explosion and (ii) makes the internal ML representation hard to understand. In this short paper, we discuss the design of a novel approach: feature-based mapping learning.

7. You Can REST Now: Automated Specification Inference and Black-Box Testing of RESTful APIs with Large Language Models

   Alix Decrop , Gilles Perrouin, Mike Papadakis , Xavier Devroey , and Pierre-Yves Schobbens

   Feb 2024

   Abs

   RESTful APIs are popular web services, requiring documentation to ease their comprehension, reusability and testing practices. The OpenAPI Specification (OAS) is a widely adopted and machine-readable format used to document such APIs. However, manually documenting RESTful APIs is a time-consuming and error-prone task, resulting in unavailable, incomplete, or imprecise documentation. As RESTful API testing tools require an OpenAPI specification as input, insufficient or informal documentation hampers testing quality. Recently, Large Language Models (LLMs) have demonstrated exceptional abilities to automate tasks based on their colossal training data. Accordingly, such capabilities could be utilized to assist the documentation and testing process of RESTful APIs. In this paper, we present RESTSpecIT, the first automated RESTful API specification inference and black-box testing approach leveraging LLMs. The approach requires minimal user input compared to state-of-the-art RESTful API inference and testing tools; Given an API name and an LLM key, HTTP requests are generated and mutated with data returned by the LLM. By sending the requests to the API endpoint, HTTP responses can be analyzed for inference and testing purposes. RESTSpecIT utilizes an in-context prompt masking strategy, requiring no model fine-tuning. Our evaluation demonstrates that RESTSpecIT is capable of: (1) inferring specifications with 85.05% of GET routes and 81.05% of query parameters found on average, (2) discovering undocumented and valid routes and parameters, and (3) uncovering server errors in RESTful APIs. Inferred specifications can also be used as testing tool inputs.

2023

1. 27th ACM International Systems and Software Product Line Conference, SPLC 2023: Welcome Message

   Paolo Arcaini , Maurice H. Ter Beek , Gilles Perrouin, Iris Reinhartz-Berger , Miguel R. Luaces , Christa Schwanninger , Shaukat Ali , Mahsa Varshosaz , and 23 more authors

   Aug 2023
2. Architectural Bad Smells for Self-Adaptive Systems: Go Runtime!

   Edilton Lima dos Santos , Pierre-Yves Schobbens , Ivan Machado , and Gilles Perrouin

   In Proceedings of the 17th International Working Conference on Variability Modelling of Software-Intensive Systems, VaMoS 2023, Odense, Denmark, January 25-27, 2023 , Jan 2023

   Abs

   Self-adaptive systems (SAS) change their behavior and structure at runtime depending on environmental changes or user requests. For this purpose, the SASs combine architectural fragments or solutions in their adaptation process. However, this process may negatively impact the system’s architectural qualities, exhibiting architectural bad smells (ABS). Current studies perform ABS detection for SAS at design time, ignoring their intrinsic runtime variability. We demonstrate that this ignorance leads to inaccurate smell detections and possibly wrong maintenance decisions. We delineate the challenges runtime variability raise on ABS detection and argue that we should analyze SAS architectures at runtime.

3. BURST: Benchmarking uniform random sampling techniques

   Mathieu Acher , Gilles Perrouin, and Maxime Cordy

   Science of Computer Programming, Mar 2023

   Abs

   BURST is a benchmarking platform for uniform random sampling (URS) techniques. Given: i) the description of a sampling space provided as a Boolean formula (DIMACS), and ii) a sampling budget (time and strength of uniformity), BURST evaluates ten samplers for scalability and uniformity. BURST measures scalability based on the time required to produce a sample, and uniformity based on the state-of-the-art and proven statistical test Barbarik. BURST is easily extendable to new samplers and offers: i) 128 feature models (for highly-configurable systems), ii) many other models mined from the artificial intelligence/satisfiability solving benchmarks. BURST envisions supporting URS assessment and design across multiple research communities.

4. Explicit or Implicit? On Feature Engineering for ML-based Variability-intensive Systems

   Paul Temple , and Gilles Perrouin

   In Proceedings of the 17th International Working Conference on Variability Modelling of Software-Intensive Systems, VaMoS 2023, Odense, Denmark, January 25-27, 2023 , Jan 2023

   Abs

   Software variability engineering benefits from Machine Learning (ML) to learn e.g., variability-Aware performance models, explore variants of interest and minimize their energy impact. As the number of applications of combining variability with ML grows, we would like to reflect on what is the core to the configuration process in software variability and inference in ML: feature engineering. These disciplines previously managed features explicitly, easing graceful combinations. Now, deep learning techniques derive automatically obscure but efficient features from data.

5. FairBayRank: A Fair Personalized Bayesian Ranker

   Armielle Noulapeu Ngaffo , Julien Albert , Benoı̂t Frénay , and Gilles Perrouin

   In 31st European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning Bruges, Belgium October 04 - 06 , Jan 2023

   Abs

   Recommender systems are data-driven models that successfully pro- vide users with personalized rankings of items (movies, books...). Meanwhile, for user minority groups, those systems can be unfair in predicting users’ expectations due to biased data. Consequently, fairness remains an open challenge in the rank- ing prediction task. To address this issue, we propose in this paper FairBayRank, a fair Bayesian personalized ranking algorithm that deals with both fairness and ranking performance requirements. FairBayRank evaluation on real-world datasets shows that it efficiently alleviates unfairness issues while ensuring high prediction performances.

6. MUPPAAL: Reducing and Removing Equivalent and Duplicate Mutants in UPPAAL

   Jaime CUARTAS GRANADA , Jesus Alexander ARANDA BUENO , Maxime Cordy , James Jerson Ortiz Vega , Gilles Perrouin, and Pierre-Yves Schobbens

   In Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2023 , Apr 2023

   Abs

   Mutation Testing (MT) is a test quality assessment technique that creates mutants by injecting artificial faults into the system and evaluating the ability of tests to distinguish these mutants. We focus on MT for safety-critical Timed Automata (TA). MT is prone to equivalent and duplicate mutants, the former having the same behaviour as the original system and the latter other mutants. Such mutants bring no value and induce useless test case executions. We propose MUPPAAL, a tool that: (1) offers a new operator reducing the occurrence of mutant duplicates; (2) an efficient bisimulation algorithm removing remaining duplicates; (3) leverages existing equivalence-avoiding mutation operators. Our experiments on four UPPAAL case studies indicate that duplicates represent up to 32% of all mutants and that the MUPPAAL bisimulation algorithm can identify them more than 99% of the time.

7. Providing command and control agility: A software product line approach

   Junier Caminha Amorim , Eduardo Lemos Rocha , Luigi Minardi , Vander Alves , Edison Pignaton de Freitas , Thiago Castro , Moussa Amrani , James Ortiz , and 2 more authors

   Expert Systems with Applications, Apr 2023

   Abs

   Command and Control (C2) is a broad concept that encompasses the coordination of individuals and organizations towards achieving a goal. However, dynamic and uncertain scenarios, such as military and disaster relief operations, present an inherent challenge to C2 activities. In such situations, plans often need to be changed in the face of unforeseen problems, and even coordination processes may be subject to variation. This dynamism increases the complexity of resource management and requires C2 Agility—i.e., the ability to respond to change in a timely and suitable fashion. Nonetheless, there is a lack of solutions to provide C2 Agility to cope with dynamic contexts. To address this problem, this work proposes a computational model of C2 Agility for a team of autonomous agents. This model describes how to combine reconfiguration of individual team members and of coordination approaches to adapt to context changes. The proposed approach leverages a typed-parameterized extension of a channel system to define the coordinating roles and responsibilities of team members. Each member is modeled as a dynamic software product line, with the inherent ability to reconfigure itself. To assess this model, a team of Unmanned Aerial Vehicles (UAV) performing a reconnaissance mission was simulated. The simulation showed that the proposed model was suitable for dealing with dynamic contexts. Particularly, metrics for the agile approach suggest improved system resilience in the face of induced perturbations, compared to non-agile C2. The obtained results with the proposed software-based simulations showed that the proposed model is useful in providing C2 Agility to the studied scenarios, making the behavior of the entities specified in the model capable of dealing with context changes.

2022

1. Abstract: Towards Assessing Architectural Smells for Self-Adaptive Systems at Runtime

   Edilton Lima dos Santos , Pierre-Yves Schobbens , and Gilles Perrouin

   Sep 2022

   Abs

   Self-adaptive systems (SAS) change their behavior and structure at runtime to answer the changes in their environment. Such systems combine different architectural fragments or solutions via feature binding/unbinding at runtime. Moreover, this combination may negatively impact the system’s architectural qualities, exhibiting architectural bad smells (ABS). These issues are challenging to detect in the code due to the combinatorial explosion of interactions amongst features. Since SAS do not document these features in their source code, design time smell detection ignores them and risks reporting smells that are different than those observed at runtime. This paper assesses this risk to understand how ABS occur at runtime for different feature combinations. We look for cyclic dependency and hub-like ABS in various runtime adaptations of two SAS, Adasim and mRubis. Our results indicate that architectural smells are feature-dependent and that their number is highly variable from one adaptation to the other. Some ABS appear in all runtime adaptations, some in only a few. We discuss the reasons behind these architectural smells for each system and draw some lessons for targeted analyses of ABS in SAS.

2. Behavioral Maps: Identifying Architectural Smells in Self-Adaptive Systems at Runtime

   Edilton Lima dos Santos , Sophie Fortz , Pierre-Yves Schobbens , and Gilles Perrouin

   In Software Architecture - 15th European Conference, ECSA 2021 Tracks and Workshops, Revised Selected Papers , Aug 2022

   Abs

   Self-adaptive systems (SAS) change their behavior and structure at runtime, depending on environmental changes and reconfiguration plans and goals. Such systems combine architectural fragments or solutions in their (re)configuration process. However, this process may negatively impact the system’s architectural qualities, exhibiting architectural bad smells (ABS). Also, some smells may appear in only particular runtime conditions. This issue is challenging to detect due to the combinatorial explosion of interactions amongst features. We initially proposed the notion of Behavioral Map to explore architectural issues at runtime. This extended study applies the Behavioral Map to analyze the ABS in self-adaptive systems at runtime. In particular, we look for Cyclic Dependency, Extraneous Connector, Hub-Like Dependency, and Oppressed Monitor ABS in various runtime adaptations in the Smart Home Environment (SHE) framework, Adasim, and mRUBiS systems developed in Java. The results indicate that runtime ABS identification is required to fully capture SAS architectural qualities because the ABS are feature-dependent, and their number is highly variable for each adaptation. We have observed that some ABS appears in all runtime adaptations, some in only a few. However, some ABS only appear in the publish-subscribe architecture, such as Extraneous Connector and Oppressed Monitor smell. We discuss the reasons behind these architectural smells for each system and motivate the need for targeted ABS analyses in SAS.

3. Featured Scents: Towards Assessing Architectural Smells for Self-Adaptive Systems at Runtime

   Edilton Lima dos Santos , Pierre-Yves Schobbens , and Gilles Perrouin

   In 2022 IEEE 19th International Conference on Software Architecture Companion, ICSA-C 2022 , Mar 2022

   Abs

   Self-adaptive systems (SAS) change their behavior and structure at runtime to answer the changes in their environment. Such systems combine different architectural fragments or solutions via feature binding/unbinding at runtime. Moreover, this combination may negatively impact the system’s architectural qualities, exhibiting architectural bad smells (ABS). These issues are challenging to detect in the code due to the combinatorial explosion of interactions amongst features. Since SAS does not document these features in their source code, design time smell detection ignores them and risks reporting smells that are different than those observed at runtime. This paper assesses this risk to understand how ABS occurs at runtime for different feature combinations. We look for cyclic dependency and hub-like ABS in various runtime adaptations of two SAS, Adasim and mRubis. Our results indicate that architectural smells are feature-dependent and that their number is highly variable from one adaptation to the other. Some ABS appear in all runtime adaptations, some in only a few. We discuss the reasons behind these architectural smells for each system and motivate the need for targeted ABS analyses in SAS.

4. Identifying Architectural Smells in Self-Adaptive Systems at Runtime

   Edilton Lima dos Santos , Sophie Fortz , Pierre-Yves Schobbens , and Gilles Perrouin

   Jun 2022

   Abs

   Self-adaptive systems (SAS) change their architecture or behavior depending on the (re)configuration plans to answer the environmental context. Moreover, the (re)configurations process may also negatively affect the system’s architectural qualities because this process combines different architectural fragments or solutions via feature binding/unbinding at runtime. Thus, Architectural Bad Smells (ABS) may emerge, implying reduced system maintainability. Also, it indicates possible design and implementation issues and helps improve the system’s quality. But, ABS is less studied in SAS. However, these studies do not analyze the impact of runtime variability on smell detection and evolution as the SAS adapts. We extend our previous work to employ the Behavioral Map to analyze the ABS in SAS at runtime. In particular, we presented the feature identification process employed to instrument the source code of the SASs to detect ABS at runtime. We targeted for Cyclic Dependency (CD), Extraneous Connector (EC), Hub-Like Dependency (HL), and Oppressed Monitor (OM) ABS in multiple runtime adaptations in the SHE [8], Adasim [9], and mRUBiS [10] systems developed in Java. Our results suggest that some ABS appear only in a specific system configuration or architecture. For instance, the EC and OM smell appear in publish-subscribe architecture, as used in SHE. Also, we observed that the type and number of ABS found in the SAS depend on the configuration analyzed at runtime. For instance, in Adasim, the BM found 9 CD and 3 HL smells in the first adaptation, but the BM found 6 CD smells in the second adaptation. We could explain this variation by binding/unbinding certain runtime features. Thus, the Behavioral Map offers interesting support for assessing the architectural qualities of a given runtime adaptation. However, instrumenting the systems for runtime ABS identification demands expertise and time because the core and variable features are not documented. This paper provides the following contributions: i) A first study to identify ABS for SAS at runtime; ii) Our analysis based on two runtime adaptations of SHE, 40 runtime adaptations of Adasim, and 16 runtime adaptations of mRUBiS, demonstrate that runtime variability affects the type and occurrence of smells found; iii) The results and scripts to process behavioral maps are available in Github.

5. IntJect: Vulnerability Intent Bug Seeding

   Benjamin PETIT , Ahmed Khanfir , Ezekiel Soremekun , Gilles Perrouin, and Michail Papadakis

   In Proceedings - 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security, QRS 2022 , Jun 2022

   Abs

   Studying and exposing software vulnerabilities is important to ensure software security, safety, and reliability. Software engineers often inject vulnerabilities into their programs to test the reliability of their test suites, vulnerability detectors, and security measures. However, state-of-the-art vulnerability injection methods only capture code syntax/patterns, they do not learn the intent of the vulnerability and are limited to the syntax of the original dataset. To address this challenge, we propose the first intent-based vulnerability injection method that learns both the program syntax and vulnerability intent. Our approach applies a combination of NLP methods and semantic-preserving program mutations (at the bytecode level) to inject code vulnerabilities. Given a dataset of known vulnerabilities (containing benign and vulnerable code pairs), our approach proceeds by employing semantic-preserving program mutations to transform the existing dataset to semantically similar code. Then, it learns the intent of the vulnerability via neural machine translation (Seq2Seq) models. The key insight is to employ Seq2Seq to learn the intent (context) of the vulnerable code in a manner that is agnostic of the specific program instance. We evaluate the performance of our approach using 1275 vulnerabilities belonging to five (5) CWEs from the Juliet test suite. We examine the effectiveness of our approach in producing compilable and vulnerable code. Our results show that IntJECT is effective, almost all (99%) of the code produced by our approach is vulnerable and compilable. We also demonstrate that the vulnerable programs generated by IntJECT are semantically similar to the withheld original vulnerable code. Finally, we show that our mutation-based data transformation approach outperforms its alternatives, namely data obfuscation and using the original data.

6. Reuse and Software Quality: 20th International Conference on Software and Systems Reuse, ICSR 2022, Montpellier, France, June 15-17, 2022, Proceedings

   Jun 2022

2021

1. A Take on Obfuscation with Ethical Adversaries

   Pieter Delobelle , Paul Temple , Gilles Perrouin, Benoit Frénay , Patrick Heymans , and Bettina Berendt

   Jun 2021
2. A Vision to identify Architectural Smells in Self-Adaptive Systems using Behavioral Maps

   Edilton Lima dos Santos , Sophie Fortz , GILLES PERROUIN , and Pierre-Yves Schobbens

   In ECSA2021 Companion Volume , Sep 2021

   Abs

   Self-adaptive systems can be implemented as Dynamic Software Product Lines (DSPLs) via dynamically enabling or disabling features at runtime based on a feature model. However, the runtime (re)configuration may negatively impact the system’s architectural qualities, exhibiting architectural bad smells. Such smells may appear in only very specific runtime conditions, and the combinatorial explosion of the number of configurations induced by features makes exhaustive analysis intractable. We are therefore targeting smell detection at runtime for one specific configuration determined through a MAPE-K loop. To support smell detection, we propose the Behavioral Map (BM) formalism to derive automatically key architectural characteristics from different sources (feature model, source code, and other deployment artifacts) and represent them in a graph. We provide identification guidelines based on the BM for four architectural smells and illustrate the approach on Smart Home Environment (SHE) DSPL.

3. BURST: A Benchmarking Platform for Uniform Random Sampling Techniques

   Mathieu Acher , GILLES PERROUIN , and Maxime Cordy

   In Proceedings of the 25th International Systems And Software Product Line Conference , Sep 2021
4. Empirical Assessment of Generating Adversarial Configurations for Software Product Lines

   Paul Temple , Gilles Perrouin, Mathieu Acher , Battista Biggio , Jean-Marc Jézéquel , and Fabio Roli

   Empirical Software Engineering, Jan 2021

   Abs

   Software product line (SPL) engineering allows the derivation ofproducts tailored to stakeholders’ needs through the setting of a large numberof configuration options.Unfortunately, options and their interactions create a huge configurationspace which is either intractable or too costly to explore exhaustively. Insteadof covering all products, machine learning (ML) approximates the set of ac-ceptable products (e.g.,successful builds, passing tests) out of a training set (asample of configurations). However, ML techniques can make prediction errorsyielding non-acceptable products wasting time, energy and other resources.We applyadversarial machine learning techniquesto the world of SPLs andcraft new configurations faking to be acceptable configurations but that arenot and vice-versa. It allows to diagnose prediction errors and take appropriateactions. We develop two adversarial configuration generators on top of state-of-the-art attack algorithms and capable of synthesizing configurations thatare both adversarial and conform to logical constraints.We empirically assess our generators within two case studies: an industrialvideo synthesizer (MOTIV) and an industry-strength, open-source Web-appconfigurator (JHipster). For the two cases, our attacks yield (up to) a 100%misclassification rate without sacrificing the logical validity of adversarial con-figurations. This work lays the foundations of a quality assurance frameworkfor ML-based SPLs.

5. Ethical Adversaries: Towards Mitigating Unfairness with Adversarial Machine Learning

   Pieter Delobelle , Paul Temple , Gilles Perrouin, Benoı̂t Frénay , Patrick Heymans , and Bettina Berendt

   SIGKDD Explorations, May 2021

   Abs

   Machine learning is being integrated into a growing number of critical systems with far-reaching impacts on society. Unexpected behaviour and unfair decision processes are coming under increasing scrutiny due to this widespread use and its theoretical considerations. Individuals, as well as organisations, notice, test, and criticize unfair results to hold model designers and deployers accountable. We offer a framework that assists these groups in mitigating unfair representations stemming from the training datasets. Our framework relies on two inter-operating adversaries to improve fairness. First, a model is trained with the goal of preventing the guessing of protected attributes’ values while limiting utility losses. This first step optimizes the model’s parameters for fairness. Second, the framework leverages evasion attacks from adversarial machine learning to generate new examples that will be misclassified. These new examples are then used to retrain and improve the model in the first step. These two steps are iteratively applied until a significant improvement in fairness is obtained. We evaluated our framework on well-studied datasets in the fairness literature - including COMPAS - where it can surpass other approaches concerning demographic parity, equality of opportunity and also the model’s utility. We investigated the trade-offs between these targets in terms of model hyperparameters and also illustrated our findings on the subtle difficulties when mitigating unfairness and highlight how our framework can assist model designers.

6. Featured Scents: Assessing Architectural Smells for Self-Adaptive Systems at Runtime

   Edilton Lima dos Santos , Pierre Yves Schobbens , and Gilles Perrouin

   Nov 2021

   Abs

   Self-adaptive systems (SAS) change their behavior and structure at runtime to answer the changes in their environment. Such systems combine different architectural fragments or solutions via feature binding/unbinding at runtime. Moreover, this combination may negatively impact the system’s architectural qualities, exhibiting architectural bad smells (ABS). These issues are challenging to detect in the code due to the combinatorial explosion of interactions amongst features. Since SAS do not document these features in their source code, design time smell detection ignores them and risks reporting spurious smells. This paper assesses this risk to understand how ABS occur at runtime for different feature combinations. We look for cyclic dependency and hub-like ABS in various runtime adaptations of two SAS, Adasim and mRubis. Our results indicate that architectural smells are feature-dependent and that their number is highly variable from one adaptation to the other. Some ABS appear in all runtime adaptations, some in only a few. We discuss the reasons behind these architectural smells for each system and draw some lessons for targeted analyses of ABS in SAS.

7. Summary of Search-based Crash Reproduction using Behavioral Model Seeding

   Pouria Derakhshanfar , Xavier Devroey , Gilles Perrouin, Andy Zaidman , and Arie Van Deursen

   In Proceedings - 2021 IEEE 14th International Conference on Software Testing, Verification and Validation, ICST 2021 , Apr 2021

   Abs

   This is an extended abstract of the article: Pouria Derakhshanfar, Xavier Devroey, Gilles Perrouin, Andy Zaidman and Arie van Deursen. 2019. Search-based crash reproduction using behavioural model seeding. In: Software Testing, Verification and Reliability (May 2020). http://doi.org/10.1002/stvr.1733.

8. VaryMinions: Leveraging RNNs to Identify Variants in Event Logs

   Sophie Fortz , Paul Temple , Xavier DEVROEY , Patrick HEYMANS , and GILLES PERROUIN

   In 5th International Workshop on Machine Learning Techniques for Software Quality Evolution , Apr 2021

   Abs

   Business processes have to manage variability in their execution, eg to deliver the correct building permit in different municipalities. This variability is visible in event logs, where sequences of events are shared by the core process (building permit authorisation) but may also be specific to each municipality. To rationalise resources (eg derive a configurable business process capturing all municipalities’ permit variants) or to debug anomalous behaviours, it is mandatory to identify to which variant a given trace belongs. This paper supports this task by training Long Short Term Memory (LSTMs) and Gated Recurrent Units (GRUs) algorithms on two datasets: a configurable municipality and a travel expenses workflow. We demonstrate that variability can be identified accurately (>87%) and discuss the challenges of learning highly entangled variants.

2020

1. An SMT-Based Concolic Testing Tool for Logic Programs

   Sophie Fortz , Fred Mesnard , Etienne Payet , Gilles Perrouin, Wim Vanhoof , and Germán Vidal

   In Functional and Logic Programming , Apr 2020

   Abs

   Concolic testing combines symbolic and concrete execution to generate test cases that achieve a good program coverage. Its benefits have been demonstrated for more than 15 years in the case of imperative programs. In this work, we present a concolic-based test generation tool for logic programs which exploits SMT-solving for constraint resolution.

2. Ethical Adversaries: Towards Mitigating Unfairness with Adversarial Machine Learning

   Pieter Delobelle , Paul Temple , Gilles Perrouin, Benoı̂t Frénay , Patrick Heymans , and Bettina Berendt

   In 1st workshop on Bias and Fairness in AI, co-located with ECMLPKDD 2020 , May 2020

   Abs

   Machine learning is being integrated into a growing number of critical systems with far-reaching impacts on society. Unexpected behaviour and unfair decision processes are coming under increasing scrutiny due to this widespread use and its theoretical considerations. Individuals, as well as organisations, notice, test, and criticize unfair results to hold model designers and deployers accountable. We offer a framework that assists these groups in mitigating unfair representations stemming from the training datasets. Our framework relies on two inter-operating adversaries to improve fairness. First, a model is trained with the goal of preventing the guessing of protected attributes’ values while limiting utility losses. This first step optimizes the model’s parameters for fairness. Second, the framework leverages evasion attacks from adversarial machine learning to generate new examples that will be misclassified. These new examples are then used to retrain and improve the model in the first step. These two steps are iteratively applied until a significant improvement in fairness is obtained. We evaluated our framework on well-studied datasets in the fairness literature – including COMPAS – where it can surpass other approaches concerning demographic parity, equality of opportunity and also the model’s utility. We also illustrate our findings on the subtle difficulties when mitigating unfairness and highlight how our framework can assist model designers.

3. Extended abstract: Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack

   Alexandre Nuttinck , Mathieu Acher , Xavier Devroey , Gilles Perrouin, Benoit Baudry , and Axel Halin

   In Proceedings of the 24th ACM Conference on Systems and Software Product Line , Oct 2020

   Abs

   This is an extended abstract of the article: Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, and Benoit Baudry. 2018. Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack. In Empirical Software Engineering (17 Jul 2018). https://doi.org/10.1007/s10664-018-9635-4.

4. MALTESQUE 2019 Workshop Summary

   Francesca Arcelli Fontana , Gilles Perrouin, Apostolos Ampatzoglou , Mathieu Acher , Bartosz Walter , Maxime Cordy , Fabio Palomba , and Xavier Devroey

   Jan 2020

   Abs

   Welcome to the third edition of the workshop on Machine Learn- ing Techniques for Software Quality Evaluation (MaLTeSQuE 2019), held in Tallinn, Estonia, August 27th, 2019, co-located with ESEC / FSE 2019. This year MALTESQUE merged with the MASES (Machine Learning and Software Engineering in Symbiosis) work- shop, co-located with the ASE 2018 conference. Ten papers from all over the world were submitted, seven of them were accepted. The program also featured a keynote by Lionel Briand on the use of machine learning to improve software testing.

5. Search-based crash reproduction using behavioural model seeding

   Pouria Derakhshanfar , Xavier DEVROEY , Gilles Perrouin, Andy Zaidman , and Arie van Deursen

   Software Testing, Verification and Reliability, May 2020

   Abs

   Search-based crash reproduction approaches assist developers during debugging by generating a test case, which reproduces a crash given its stack trace. One of the fundamental steps of this approach is creating objects needed to trigger the crash. One way to overcome this limitation is seeding: using information about the application during the search process. With seeding, the existing usages of classes can be used in the search process to produce realistic sequences of method calls, which create the required objects. In this study, we introduce behavioural model seeding: a new seeding method that learns class usages from both the system under test and existing test cases. Learned usages are then synthesized in a behavioural model (state machine). Then, this model serves to guide the evolutionary process. To assess behavioural model seeding, we evaluate it against test seeding (the state-of-the-art technique for seeding realistic objects) and no seeding (without seeding any class usage). For this evaluation, we use a benchmark of 122 hard-to-reproduce crashes stemming from six open-source projects. Our results indicate that behavioural model seeding outperforms both test seeding and no seeding by a minimum of 6% without any notable negative impact on efficiency.

6. STARS: Software Technology for Adaptable and Reusable Systems PhD Research Project

   Edilton Lima dos Santos , Gilles Perrouin, and Pierre Yves Schobbens

   In Proceedings - VaMoS 2020 , Feb 2020

   Abs

   Dynamically Adaptive Systems (DAS) modify their behaviours in response to (sometimes unpredictable) changes in their environment or to the evolution of their own abilities (sensors and actuators). To support adaptation, a reference architecture (such as the MAPE-K model) is paramount. Yet, this is not sufficient as challenges concerning the fine-grained variability management and testability remain. The STARS Ph.D. project aims at developing a variability and context-aware architectural model for DAS that particularly takes into account testability. In particular, we want to adapt tests at runtime in order to assess and prevent the impact of inappropriate adaptations.

2019

1. AbsCon: A Test Concretizer for Model-based Testing

   Jérémy Vanhecke , Xavier Devroey , and Gilles Perrouin

   In Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2019 , Apr 2019

   Abs

   Test definition and execution is an essential but time-consuming task during system development. To speed up the process, model-based testing and other related approaches propose to generate/write abstract test cases and to concretize them using either transformations, an adapter, or a mixture of the two. QTaste is an industrial data-driven test case definition and execution environment used to perform black-box testing on various kinds of systems. QTaste’s test cases are manually written in Python and use an adapter, called test API, to execute operations on the System Under Test (SUT) interfaces. In this paper, we describe AbsCon (Abstract test case Concretizer), a plugin used to generate test cases executable in QTaste based on their definition: i.e., sequences of abstract actions and assertions. AbsCon uses programmer friendly mappings (written in Python) for the SUT’s interfaces, actions, and assertions, to generate standard test cases in QTaste format. Rather than having a complete model-based testing transformation chain, the plugin is bridging the gap between existing model-based test case generation tools and an industrial test case execution system using a mix of transformation and adaptation.

2. A Decade of Featured Transition Systems

   Maxime Cordy , Xavier Devroey , Axel Legay , Gilles Perrouin, Andreas Classen , Patrick Heymans , Pierre-Yves Schobbens , and Jean-François Raskin

   Oct 2019

   Abs

   Variability-intensive systems (VIS) form a large and heterogeneous class of systems whose behaviour can be modified by enabling or disabling predefined features. Variability mechanisms allows the adaptation of software to the needs of their users and the environment. However, VIS verification and validation (V&V) is challenging: the combinatorial explosion of the number of possible behaviours and undesired feature interactions are amongst such challenges. To tackle them, Featured Transitions Systems (FTS) were proposed a decade ago to model and verify the behaviours of VIS. In an FTS, each transition is annotated with a combination of features determining which variants can execute it. An FTS can model all possible behaviours of a given VIS. This compact model enabled us to create efficient V&V algorithms taking advantage of the behaviours shared amongst features resulting in a reduction of the V&V effort by several orders of magnitude. In this paper, we will cover the formalism, its applications and sketch promising research directions.

3. Customizing Adversarial Machine Learning to test Deep Learning techniques

   Paul Temple , Gilles Perrouin, Benoı̂t Frénay , and Pierre-Yves Schobbens

   May 2019
4. Editorial to the theme section on model-based testing

   Mike Papadakis , Shaukat Ali , and Gilles Perrouin

   Software and Systems Modeling, Apr 2019

   Abs

   This theme on model-based testing (MBT) was organized in the context of advances in model-based testing (A-MOST) workshop series. Now in its fourteenth edition, this workshop covers all aspects of MBT from theoretical developments to industrial implementations. Following the twelfth edition in Chicago (2016), we invited the MBT community to submit their latest and finest research in the field via an open call. We selected seven articles that are presented in this theme section.

5. Preface to the thirteenth workshop on variability modelling of software-intensive systems

   Gilles Perrouin, and Danny Weyns

   In Proceedings of the 13th International Workshop on Variability Modelling of Software-Intensive Systems, VAMOS 2019 , Feb 2019
6. Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack

   Axel Halin , Alexandre Nuttinck , Mathieu Acher , Xavier Devroey , Gilles Perrouin, and Benoit Baudry

   Empirical Software Engineering, Apr 2019

   Abs

   Many approaches for testing configurable software systems start from the same assumption: it is impossible to test all configurations. This motivated the definition of variability-aware abstractions and sampling techniques to cope with large configuration spaces. Yet, there is no theoretical barrier that prevents the exhaustive testing of all configurations by simply enumerating them if the effort required to do so remains acceptable. Not only this: we believe there is a lot to be learned by systematically and exhaustively testing a configurable system. In this case study, we report on the first ever endeavour to test all possible configurations of the industry-strength, open source configurable software system JHipster, a popular code generator for web applications. We built a testing scaffold for the 26,000+ configurations of JHipster using a cluster of 80 machines during 4 nights for a total of 4,376 hours (182 days) CPU time. We find that 35.70% configurations fail and we identify the feature interactions that cause the errors. We show that sampling strategies (like dissimilarity and 2-wise): (1) are more effective to find faults than the 12 default configurations used in the JHipster continuous integration; (2) can be too costly and exceed the available testing budget. We cross this quantitative analysis with the qualitative assessment of JHipster’s lead developers.

7. Towards quality assurance of software product lines with adversarial configurations

   Paul Temple , Mathieu Acher , Gilles Perrouin, Battista Biggio , Jean Marc Jézéquel , and Fabio Roli

   In SPLC 2019 - 23rd International Systems and Software Product Line Conference , Sep 2019

   Abs

   Software product line (SPL) engineers put a lot of effort to ensure that, through the setting of a large number of possible configuration options, products are acceptable and well-tailored to customers’ needs. Unfortunately, options and their mutual interactions create a huge configuration space which is intractable to exhaustively explore. Instead of testing all products, machine learning is increasingly employed to approximate the set of acceptable products out of a small training sample of configurations. Machine learning (ML) techniques can refine a software product line through learned constraints and a priori prevent non-acceptable products to be derived. In this paper, we use adversarial ML techniques to generate adversarial configurations fooling ML classifiers and pinpoint incorrect classifications of products (videos) derived from an industrial video generator. Our attacks yield (up to) a 100% misclassification rate and a drop in accuracy of 5%. We discuss the implications these results have on SPL quality assurance.

8. Uniform Sampling of SAT Solutions for Configurable Systems: Are We There Yet?

   Quentin Plazar , Mathieu Acher , Gilles Perrouin, Xavier Devroey , and Maxime Cordy

   In Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019 , Apr 2019

   Abs

   Uniform or near-uniform generation of solutions for large satisfiability formulas is a problem of theoretical and practical interest for the testing community. Recent works proposed two algorithms (namely UniGen and QuickSampler) for reaching a good compromise between execution time and uniformity guarantees, with empirical evidence on SAT benchmarks. In the context of highly-configurable software systems (e.g., Linux), it is unclear whether UniGen and QuickSampler can scale and sample uniform software configurations. In this paper, we perform a thorough experiment on 128 real-world feature models. We find that UniGen is unable to produce SAT solutions out of such feature models. Furthermore, we show that QuickSampler does not generate uniform samples and that some features are either never part of the sample or too frequently present. Finally, using a case study, we characterize the impacts of these results on the ability to find bugs in a configurable system. Overall, our results suggest that we are not there: more research is needed to explore the cost-effectiveness of uniform sampling when testing large configurable systems.

9. Welcome from the chairs

   Francesca Arcelli Fontana , Gilles Perrouin, Apostolos Ampatzoglou , Mathieu Acher , Bartosz Walter , Maxime Cordy , Fabio Palomba , and Xavier Devroey

   Aug 2019

2018

1. Model-based mutant equivalence detection using automata language equivalence and simulations

   Xavier Devroey , Gilles Perrouin, Mike Papadakis , Axel Legay , Pierre-Yves Schobbens , and Patrick Heymans

   Journal of Systems and Software, Jul 2018

   Abs

   Mutation analysis is a popular technique for assessing the strength of test suites. It relies on the mutation score, which indicates their fault-revealing potential. Yet, there are mutants whose behaviour is equivalent to the original system, wasting analysis resources and preventing the satisfaction of a 100% mutation score. For finite behavioural models, the Equivalent Mutant Problem (EMP) can be transformed to the language equivalence problem of non-deterministic finite automata for which many solutions exist. However, these solutions are quite expensive, making computation unbearable when used for tackling the EMP. In this paper, we report on our assessment of a state-of-the-art exact language equivalence tool and two heuristics we proposed. We used 12 models, composed of (up to) 15,000 states, and 4,710 mutants. We introduce a random and a mutation-biased simulation heuristics, used as baselines for comparison. Our results show that the exact approach is often more than ten times faster in the weak mutation scenario. For strong mutation, our biased simulations can be up to 1,000 times faster for models larger than 300 states, while limiting the error of misclassifying non-equivalent mutants as equivalent to 8% on average. We therefore conclude that the approaches can be combined for improved efficiency.

2. Model-based Mutation Operators for Timed Systems: A Taxonomy and Research Agenda

   James Jerson Ortiz Vega , Gilles Perrouin, Moussa Amrani , and Pierre-Yves Schobbens

   Jul 2018

   Abs

   Mutation testing relies on the principle of artificially injecting faults in systems to create mutants, in order to either assess the sensitivity of existing test suites, or generate test cases that are able to find real faults. Mutation testing has been employed in a variety of application areas and at various levels of abstraction (code and models). In this paper, we focus on model-based mutation testing for timed systems. In order to cartography the field, we provide a taxonomy of mutation operators and discuss their usages on various formalisms, such as timed automata or synchronous languages. We also delineate a research agenda for the field addressing mutation costs, the impact of delays in operators specification and mutation equivalence.

3. Software Testing Research

   Gilles Perrouin

   Jul 2018
4. Welcome from the MASES 2018 chairs

   Gilles Perrouin, Mathieu Acher , Maxime Cordy , and Xavier Devroey

   Proceedings of the International Workshop on Machine Learning and Software Engineering in Symbiosis, co-located with ASE 2018, Sep 2018

2017

1. Automata Language Equivalence vs. Simulations for Model-based Mutant Equivalence: An Empirical Evaluation

   Xavier Devroey , Gilles Perrouin, Mike Papadakis , Axel Legay , Pierre Schobbens , and Patrick Heymans

   In Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation, ICST 2017 , May 2017

   Abs

   Mutation analysis is a popular test assessment method. It relies on the mutation score, which indicates how many mutants are revealed by a test suite. Yet, there are mutants whose behaviour is equivalent to the original system, wasting analysis resources and preventing the satisfaction of the full (100%) mutation score. For finite behavioural models, the Equivalent Mutant Problem (EMP) can be addressed through language equivalence of non-deterministic finite automata, which is a well-studied, yet computationally expensive, problem in automata theory. In this paper, we report on our preliminary assessment of a state-of-the-art exact language equivalence tool to handle the EMP against 3 models of size up to 15,000 states on 1170 mutants. We introduce random and mutation-biased simulation heuristics as baselines for comparison. Results show that the exact approach is often more than ten times faster in the weak mutation scenario. For strong mutation, our biased simulations are faster for models larger than 300 states. They can be up to 1,000 times faster while limiting the error of misclassifying non-equivalent mutants as equivalent to 10% on average. We therefore conclude that the approaches can be combined for improved efficiency.

2. Engineering Configuration Graphical User Interfaces from Variability Models

   Quentin Boucher , Gilles Perrouin, Jean-Marc Davril , and Patrick Heymans

   May 2017

   Abs

   In the past, companies produced large amounts of products through mass production lines. Advantages of such an approach are reduced production costs and time-to-market. While it is (still) appropriate for some goods like food or household items, customer preferences evolve to customised products. In a more and more competitive environment, product customisation is taken to the extreme by companies in order to gain market share. Companies provide customisation tools, more commonly called product configurators, to assist their staff and customers in deciding upon the characteristics of the product to be delivered. Our experience reveals that some existing configurators are implemented in an ad-hoc fashion. This is especially cumbersome when numerous and non-trivial constraints have to be dealt with. For instance, we have observed in two industrial cases that relationships between configuration options are hard-coded and mixed with GUI code. As constraints are scattered in the source code, severe maintenance issues occur.In this chapter, we present a pragmatic and model-driven way to generate configuration GUIs. We rely on feature models to represent and reason about the configuration options and their complex relationships. Once feature models have been elaborated, there is still a need to produce a GUI, including the integration with underlying reasoning mechanisms to control and update the GUI elements. We present a model-view-presenter architecture to design configurators, which separates concerns between a feature model (configuration option modelling), its associated solver (automated reasoning support) and the presentation of the GUI. To fill the gap between feature models and configuration GUIs, the various constructs of the feature model formalism are rendered as GUI elements through model transformations. Those transformations can be parametrised through beautification and view languages to derive specific configuration GUIs. We illustrate our approach on an IPv6 addressing plan configurator.

3. Feature-based Elicitation of Cognitively Efficient Visualizations for SPL Configurations

   Céline Sauvage-Thomase , Nicolas Biri , Gilles Perrouin, Nicolas Genon , and Patrick Heymans

   May 2017

   Abs

   Configuring a SPL is a cognitively difficult activity that requires a deepunderstanding of the features and their constraints to be performed effectively.To this end, SPL configurators have been equipped with various visualizations toassist users in their tasks. However, there are many ways to visualize data: theprocess of associating an efficient visualization to a given (configuration) task is neitherwell-understood nor systematically applied, resulting in confusingvisualizations yielding configuration errors. In this chapter, we offer such aprocess, based on theories of the visualization community for data representation. The first step consists in choosing the data to be visualized. Thisselection induces restrictions on the types of visualization that are thencomputed based on the data characteristics and best practices from semiology andvisual languages. Designers can then select an efficientvisualization for the intended task. Our process is supported by feature modelsand FAMILIAR to merge and constrain the set of applicable visualizations.

4. On Featured Transition Systems

   Axel Legay , Gilles Perrouin, Xavier Devroey , Maxime Cordy , Pierre-Yves Schobbens , and Patrick Heymans

   In SOFSEM 2017: Theory and Practice of Computer Science , Jan 2017

   Abs

   Software Product Lines (SPLs) are families of similar software products built from a common set of features. As the number of products of an SPL is potentially exponential in the number of its features, analysing SPLs is harder than for single software. In this invited paper, we synthesise six years of efforts in alleviating SPL verification and testing issues. To this end, we introduced Featured Transition Systems (FTS) as a compact behavioural model for SPLs. Based on this formalism, we designed verification algorithms and tools allowing to check temporal properties on FTS, thereby assessing the correct behaviour of all the SPL products. We also used FTS to define test coverage and generation techniques for model-driven SPLs. We also successfully employed the formalism in order to foster mutation analysis. We conclude with future directions on the development of FTS for SPL analysis.

5. On quantitative requirements for product lines

   Axel Legay , and Gilles Perrouin

   In Proceedings - VaMoS 2017 , Feb 2017

   Abs

   Software Product Line Engineering (SPLE) aims at developing a large number of software systems that share a common and managed set of features [5]. In the past years, it has been an active area in both research and industry. SPLE aims at improving productivity and reducing the time, effort, and cost required to develop a family of products (also called variants). The key point to achieve this goal is to manage the variability among various products of a Software Product Line (SPL). Variability is commonly expressed in terms of features, i.e., units of difference between software products. A product can thus be viewed as a set of features. Dependencies between features are typically represented in a Feature Model (FM) [11], whose ultimate purpose is to define which combinations of features (that is, which products) are valid [16]. Behavior of both the features and the core behavior (i.e., the behavior shared by all products in the line) may be represented by (variants of) state machines [3, 13].

6. Statistical prioritization for software product line testing: an experience report

   Xavier Devroey , Gilles Perrouin, Maxime Cordy , Hamza Samih , Axel Legay , Pierre Yves Schobbens , and Patrick Heymans

   Software and Systems Modeling, Feb 2017

   Abs

   Software product lines (SPLs) are families of software systems sharing common assets and exhibiting variabilities specific to each product member of the family. Commonalities and variabilities are often represented as features organized in a feature model. Due to combinatorial explosion of the number of products induced by possible features combinations, exhaustive testing of SPLs is intractable. Therefore, sampling and prioritization techniques have been proposed to generate sorted lists of products based on coverage criteria or weights assigned to features. Solely based on the feature model, these techniques do not take into account behavioural usage of such products as a source of prioritization. In this paper, we assess the feasibility of integrating usage models into the testing process to derive statistical testing approaches for SPLs. Usage models are given as Markov chains, enabling prioritization of probable/rare behaviours. We used featured transition systems, compactly modelling variability and behaviour for SPLs, to determine which products are realizing prioritized behaviours. Statistical prioritization can achieve a significant reduction in the state space, and modelling efforts can be rewarded by better automation. In particular, we used MaTeLo, a statistical test cases generation suite developed at ALL4TEC. We assess feasibility criteria on two systems: Claroline, a configurable course management system, and Sferion™, an embedded system providing helicopter landing assistance.

7. Testing Variability-Intensive Systems

   Gilles Perrouin, Xavier Devroey , and Maxime Cordy

   In SPLC 2017 - 21st International Systems and Software Product Line Conference, Proceedings , Sep 2017

   Abs

   Cost-effective techniques to test software in the presence of variability are popular in academia but are not systematically applied in practice yet. This half-day tutorial offers an overview of the state of the art of some of the hottest topics in the field such as (dis)similarity & search-based testing, model-based testing or mutation analysis. We will present the most significant results obtained during the last five years ranging from conceptual foundations to readily usable tools. In particular, we will rely on VIBeS, a model-based open source framework we developed to test product lines behaviour. We will illustrate all these concepts on JHipster, a variability-intensive web development stack.

8. Towards Security-aware Mutation Testing

   Thomas Loise , Xavier Devroey , Gilles Perrouin, Mike Papadakis , and Patrick Heymans

   In Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017 , Mar 2017

   Abs

   Mutation analysis forms a popular software analysis technique that has been demonstrated to be useful in supporting multiple software engineering activities. Yet, the use of mutation analysis in tackling security issues has received little attention. In view of this, we design security aware mutation operators to support mutation analysis. Using a known set of common security vulnerability patterns, we introduce 15 security-aware mutation operators for Java. We then implement them in the PIT mutation engine and evaluate them. Our preliminary results demonstrate that standard PIT operators are unlikely to introduce vulnerabilities similar to ours. We also show that our security-aware mutation operators are indeed applicable and prevalent on open source projects, providing evidence that mutation analysis can support security testing activities.

9. Yo Variability! JHipster: A Playground for Web-Apps Analyses

   Axel Halin , Alexandre Nuttinck , Mathieu Acher , Xavier Devroey , Gilles Perrouin, and Patrick Heymans

   In Proceedings - VaMoS 2017 , Feb 2017

   Abs

   Though variability is everywhere, there has always been a shortage of publicly available cases for assessing variability-aware tools and techniques as well as supports for teaching variability-related concepts. Historical software product lines contains industrial secrets their owners do not want to disclose to a wide audience. The open source community contributed to large-scale cases such as Eclipse, Linux kernels, or web-based plugin systems (Drupal, WordPress). To assess accuracy of sampling and prediction approaches (bugs, performance), a case where all products can be enumerated is desirable. As configuration issues do not lie within only one place but are scattered across technologies and assets, a case exposing such diversity is an additional asset. To this end, we present in this paper our efforts in building an explicit product line on top of JHipster, an industrial open-source Web-app configurator that is both manageable in terms of configurations (approx. 163,000) and diverse in terms of technologies used. We present our efforts in building a variability-aware chain on top of JHipster’s configurator and lessons learned using it as a teaching case at the University of Rennes. We also sketch the diversity of analyses that can be performed with our infrastructure as well as early issues found using it. Our long term goal is both to support students and researchers studying variability analysis and JHipster developers in the maintenance and evolution of their tools.

2016

1. A Complexity Tale: Web Configurators

   Gilles Perrouin, Mathieu Acher , Jean-Marc Davril , Axel Legay , and Patrick Heymans

   In IEEE/ACM 1st International Workshop on Variability and Complexity in Software Design (VACE) , Feb 2016

   Abs

   Online configurators are basically everywhere. From physical goods (cars, clothes) to services (cloud solutions, insurances, etc.) such configurators have pervaded many areas of everyday life, in order to provide the customers products tailored to their needs. Being sometimes the only interfaces between product suppliers and consumers, much care has been devoted to the HCI aspects of configurators, aiming at offering an enjoyable buying experience. However, at the backend, the management of numerous and complex configuration options results from ad-hoc process rather than a systematic variability-aware engineering approach. We present our experience in analysing web configurators and formalising configuration options in terms of feature models or product configuration matrices. We also consider behavioural issues and perspectives on their architectural design.

2. Featured Model-based Mutation Analysis

   Xavier Devroey , Gilles Perrouin, Mike Papadakis , Axel Legay , Pierre Schobbens , and Patrick Heymans

   In Proceedings of the 38th international conference on Software Engineering , May 2016

   Abs

   Model-based mutation analysis is a powerful but expensive testing technique. We tackle this problem by proposing an optimization technique that drastically speeds up the mutant execution process. Central to this approach is the Featured Mutant Model, a modeling framework for mutation analysis inspired by the software product line paradigm. It uses behavioral variability models, viz., Featured Transition Systems, which enable the optimized generation, configuration and execution of mutants. We provide results, based on models with thousands of transitions, suggesting that our technique is fast and scalable. We found that it outperforms previous approaches by several orders of magnitude and that it makes higher-order mutation practically applicable.

3. Featured Model Types: Towards Systematic Reuse in Modelling Language Engineering

   Gilles Perrouin, Moussa Amrani , Mathieu Acher , Axel Legay , and Pierre-Yves Schobbens

   In 8th Workshop on Modelling in Software Engineering (MiSE’2016) , May 2016

   Abs

   By analogy with software product reuse, the ability to reuse (meta)models and model transformations is key to achieve better quality and productivity. To this end, various opportunistic reuse techniques have been developed, such as higher-order transformations, metamodel adaptation, and model types. However, in contrast to software product development that has moved to systematic reuse by adopting (model-driven) software product lines, we are not quite there yet for modelling languages, missing economies of scope and automation opportunities. Our vision is to transpose the product line paradigm at the metamodel level, where reusable assets are formed by metamodel and transformation fragments and ?products? are reusable language building blocks (model types). We introduce featured model types to concisely model variability amongst metamodelling elements, enabling configuration, automated analysis, and derivation of tailored model types. We provide a wish list of software engineering activities to work with featured model types.FSE

4. Message from the A-MOST 2016 Chairs

   Mike Papadakis , Gilles Perrouin, and Shaukat Ali

   In 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW) , Apr 2016
5. Search-based Similarity-driven Behavioural SPL Testing

   Xavier Devroey , Gilles Perrouin, Axel Legay , Pierre-Yves Schobbens , and Patrick Heymans

   In Proceedings of the Tenth International Workshop on Variability Modelling of Software-intensive Systems , Jan 2016

   Abs

   Dissimilar test cases have been proven to be effective to reveal faults in software systems. In the Software Product Line (SPL) context, this criterion has been applied successfully to mimic combinatorial interaction testing in an efficient and scalable manner by selecting and prioritising most dissimilar configurations of feature models using evolutionary algorithms. In this paper, we extend dissimilarity to behavioural SPL models (FTS) in a search-based approach, and evaluate its effectiveness in terms of product and fault coverage. We investigate different distances as well as as single-objective algorithms, (dissimilarity on actions, random, all-actions). Our results on four case studies show the relevance of dissimilarity-based test generation for behavioural SPL models, especially on the largest case-study where no other approach can match it.

6. Unlocking visual understanding: Towards effective keys for diagrams

   Nicolas Genon , Gilles Perrouin, Xavier Le Pallec , and Patrick Heymans

   In Conceptual Modeling - 35th International Conference, ER 2016, Proceedings , Jan 2016

   Abs

   Diagrams are (meant to be) effective communication supports to convey information to stakeholders. Being communication supports, they have to be quickly and accurately understood. To enable immediateness, many disciplines such as cartography rely on keys, which categorise diagram symbols and bind them to their meaning. Software engineering extensively relies on visual languages such as UML to communicate amongst the many stakeholders involved in information systems’ life-cycle. Yet, keys are barely used in these diagrams, hindering (immediate) understanding and limiting it to language experts. We provide a disciplined approach to design effective keys, by adapting graphic semiology theory and cartographers’ know-how to software diagrams.We illustrate our method on a UML class diagram. Designing effective keys raises questions about the concerns and tasks to be addressed by the diagram, and even, reveals issues about the language itself.

7. Variability and Complexity in Software Design: Towards a Research Agenda

   Matthias Galster , Uwe Zdun , Danny Weyns , Rick Rabiser , Bo Zhang , Michael Goedicke , and Gilles Perrouin

   ACM SIGSOFT Software Engineering Notes, Nov 2016

   Abs

   Many of todayâs software systems accommodate different usage and deployment scenarios. Intentional and unintentional variability in functionality or quality attributes (e.g., performance) of software significantly increases the complexity of the problem and design space of those systems. The complexity caused by variability becomes increasingly difficult to handle due to the increasing size of software systems, new and emerging application domains, dynamic operating conditions under which software systems have to operate, fast moving and highly competitive markets, and more powerful and versatile hardware. This paper reports results of the first International Workshop on Variability and Complexity in Software Design that brought together researchers and engineers interested in the topic of complexity and variability. It also outlines directions the field might move in the future.

2015

1. Covering SPL Behaviour with Sampled Configurations: An Initial Assessment

   Xavier Devroey , Gilles Perrouin, Axel Legay , Pierre-Yves Schobbens , and Patrick Heymans

   In Proceedings of the Ninth International Workshop on Variability Modelling of Software-intensive Systems , Nov 2015

   Abs

   Structural approaches to Software Product Lines (SPL) testing (such as pairwise testing) have gained momentum as they are able to scale to larger SPLs described as feature diagrams (FD). However, these methods are agnostic with respect to behaviour: the sampled configurations have thus no reason to satisfy any given behavioural criterion. In this paper, we investigate the behavioural coverage of two structural testing criteria: pairwise and similarity. To do so, we modelled four SPLs in terms of feature diagrams and associated featured transitions systems (FTSs). We then computed state, action and transition coverage for a set of generated configurations. Preliminary results indicate that for relatively small variability models with few cross-tree constraints, structural coverage-driven tools tend to cover large parts of behaviour with less than 8 configurations. Though structural coverage cannot be used directly as a replacement for behavioural driven SPL test generation, opportunities to mix structural and behavioural coverage for efficient and effective SPL testing do exist.

2. Message from the chairs - A MOST 2015

   Mike Papadakis , Levi Lucio , and Gilles Perrouin

   In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2015 - Proceedings , May 2015
3. Poster: VIBeS, Transition System Mutation Made Easy

   Xavier Devroey , Gilles Perrouin, Pierre Schobbens , and Patrick Heymans

   In Proceedings - International Conference on Software Engineering , Aug 2015

   Abs

   Mutation testing is an established technique used to evaluate the quality of a set of test cases. As model-based testing took momentum, mutation techniques were lifted to the model level. However, as for code mutation analysis, assessing test cases on a large set of mutants can be costly. In this paper, we introduce the Variability-Intensive Behavioural teSting (VIBeS) framework. Relying on Featured Transition Systems (FTSs), we represent all possible mutants in a single model constrained by a feature model for mutant (in)activation. This allow to assess all mutants in a single test case execution. We present VIBeS implementation steps and the DSL we defined to ease model-based mutation analysis.

4. SPLat 2015 Second international workshop on software product line analysis tools

   Gilles Perrouin, and Axel Legay

   In Proceedings - 19th International Software Product Line Conference, SPLC 2015 , Jul 2015

   Abs

   SPLat 2015 workshop aims to provide a forum where various approaches to formal analysis and testing of variabilityintensive systems can be presented, evaluated and discussed. In particular, the workshop tries to identify commonalities and variabilities regarding the choice of underlying concepts that capture variability as well as strengths and weaknesses of approaches in their effort to defeat combinatorial explosion. The long term goal of the workshop is to provide guidance to practitioners on where and when to use the aforementioned techniques while validating variability-intensive systems.

2014

1. Abstract test case generation for behavioural testing of software product lines

   Xavier Devroey , Gilles Perrouin, and Pierre-Yves Schobbens

   In ACM International Conference Proceeding Series , Sep 2014

   Abs

   In Model Based Testing (MBT), test cases are generated automatically from a partial representation of expected behaviour of the System Under Test (SUT) (i.e., the model). For most industrial systems, it is impossible to generate all the possible test cases from the model. The test engineer recourse to generation algorithms that maximize a given coverage criterion, a metric indicating the percentage of possible behaviours of the SUT covered by the test cases. Our previous work redefined classical Transition Systems (TSs) criteria for SPLs, using Featured Transition Systems (FTSs), a mathematical structure to compactly represent the behaviour of a SPL, as model for test case generation. In this paper, we provide one all-states coverage driven generation algorithm and discuss its scalability and efficiency with respect to random generation. All-states and random generation are compared on fault-seeded FTSs.

2. A Variability Perspective of Mutation Analysis

   Xavier Devroey , Gilles Perrouin, Maxime Cordy , Mike Papadakis , Axel Legay , and Pierre-Yves Schobbens

   In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering , Nov 2014

   Abs

   Mutation testing is an effective technique for either improving or generating fault-finding test suites. It creates defective or incorrect program artifacts of the program under test and evaluates the ability of test suites to reveal them. Despite being effective, mutation is costly since it requires assessing the test cases with a large number of defective artifacts. Even worse, some of these artifacts are behaviourally "equivalent" to the original one and hence, they unnecessarily increase the testing effort. We adopt a variability perspective on mutation analysis. We model a defective artifact as a transition system with a specific feature selected and consider it as a member of a mutant family. The mutant family is encoded as a Featured Transition System, a compact formalism initially dedicated to model-checking of software product lines. We show how to evaluate a test suite against the set of all candidate defects by using mutant families. We can evaluate all the considered defects at the same time and isolate some equivalent mutants. We can also assist the test generation process and eciently consider higher-order mutants.

3. Bypassing the combinatorial explosion: Using similarity to generate and prioritize t-wise test configurations for software product lines

   Christopher Henard , Mike Papadakis , Gilles Perrouin, Jacques Klein , Patrick Heymans , and Yves Le Traon

   IEEE Transactions on Software Engineering, Jul 2014

   Abs

   Large Software Product Lines (SPLs) are common in industry, thus introducing the need of practical solutions to test them. To this end, t-wise can help to drastically reduce the number of product configurations to test. Current t-wise approaches for SPLs are restricted to small values of t. In addition, these techniques fail at providing means to finely control the configuration process. In view of this, means for automatically generating and prioritizing product configurations for large SPLs are required. This paper proposes (a) a search-based approach capable of generating product configurations for large SPLs, forming a scalable and flexible alternative to current techniques and (b) prioritization algorithms for any set of product configurations. Both these techniques employ a similarity heuristic. The ability of the proposed techniques is assessed in an empirical study through a comparison with state of the art tools. The comparison focuses on both the product configuration generation and the prioritization aspects. The results demonstrate that existing t-wise tools and prioritization techniques fail to handle large SPLs. On the contrary, the proposed techniques are both effective and scalable. Additionally, the experiments show that the similarity heuristic can be used as a viable alternative to t-wise. \textcopyright 1976-2012 IEEE.

4. Coverage criteria for behavioural testing of software product lines

   Xavier Devroey , Gilles Perrouin, Axel Legay , Maxime Cordy , Pierre-Yves Schobbens , and Patrick Heymans

   In Proceedings of the 6th International Symposium on Leveraging Applications of Formal Methods, Verification and Validation (ISOLA) , Nov 2014

   Abs

   Featured Transition Systems (FTS) is a mathematical structure to represent the behaviour of software product line in a concise way. The combination of the well-known transition systems approach to formal behavioural modelling with feature expressions was pivotal to the design of efficient verification approaches. Such approaches indeed avoid to consider products’ behaviour independently, leading to often exponential savings. Building on this successful structure, we lay the foundations of model-based testing approach to SPLs. We define several FTS-aware coverage criteria and report on our experience combining FTS with usage-based testing for configurable websites.

5. State Machine Flattening: Mapping Study and Assessment

   Xavier Devroey , Gilles Perrouin, Maxime Cordy , Axel Legay , Pierre-Yves Schobbens , and Patrick Heymans

   In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops , Mar 2014

   Abs

   State machine formalisms equipped with hierarchy and parallelism allow to compactly model complex system behaviours. Such models can then be transformed into executable code or inputs for model-based testing and verification techniques. Generated artifacts are mostly flat descriptions of system behaviour. Flattening is thus an essential step of these transformations. To assess the importance of flattening, we have defined and applied a systematic mapping process and 30 publications were finally selected. However, it appeared that flattening is rarely the sole focus of the publications and that care devoted to the description and validation of flattening techniques varies greatly. Preliminary assessment of associated tool support indicated limited tool availability and scalability on challenging models. We see this initial investigation as a first step towards generic flattening techniques and scalable tool support, cornerstones of reliable model-based behavioural development.

6. Towards a systematic approach for cognitively efficient configuration visualizations

   Céline Sauvage-Thomase , Nicolas Biri , Gilles Perrouin, and Patrick Heymans

   Dec 2014

   Abs

   In Software Product Line Engineering, the configuration process consists in deriving products from a variability model, usually a feature model. Configuration is difficult for stakeholders because of numerous features to choose and complex constraints relating them. If analysis tools assist in getting consistent products, they do not improve the cognitive effort to be carried out by stakeholders. Data visualization techniques are known to be helpful in this respect. However, choosing the right amongst thousands of visualizations available is often done in an ad-hoc way. In this paper, we present a systematic approach which, from a study of the data relevant to the configuration process, leads us to select an appropriate visualization for understanding configuration decisions.

7. Towards Statistical Prioritization for Software Product Lines Testing

   Xavier Devroey , Maxime Cordy , Gilles Perrouin, Pierre-Yves Schobbens , Axel Legay , and Patrick Heymans

   In Proceedings of the Eighth International Workshop on Variability Modelling of Software-Intensive Systems , Jan 2014

   Abs

   Software Product Lines (SPL) are inherently difficult to test due to the combinatorial explosion of the number of products to consider. To reduce the number of products to test, sampling techniques such as combinatorial interaction testing have been proposed. They usually start from a feature model and apply a coverage criterion (e.g. pairwise feature interaction or dissimilarity) to generate tractable, fault-finding, lists of configurations to be tested. Prioritization can also be used to sort/generate such lists, optimizing coverage criteria or weights assigned to features. However, current sampling/prioritization techniques barely take product behavior into account. We explore how ideas of statistical testing, based on a usage model (a Markov chain), can be used to extract configurations of interest according to the likelihood of their executions. These executions are gathered in featured transition systems, compact representation of SPL behavior. We discuss possible scenarios and give a prioritization procedure illustrated on an example.

8. VIBeS: Variability Intensive system Behavioural teSting

   Xavier Devroey , and Gilles Perrouin

   Oct 2014

   Abs

   VIBeS aims at providing model-driven testing tools working on Featured Transition Systems (FTSs) in order to perform: test-case selection, test-case prioritization, prioritization, test-case execution, mutation testing, etc.

2013

1. Assessing software product line testing via model-based mutation: An application to similarity testing

   Christopher Henard , Mike Papadakis , Gilles Perrouin, Jacques Klein , and Yves Le Traon

   In Proceedings - IEEE 6th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2013 , Sep 2013

   Abs

   Needs for mass customization and economies of scale have pushed engineers to develop Software Product Lines (SPLs). SPLs are families of products sharing commonalities and exhibiting differences, built by reusing software assets abstractly represented by features. Feature models describe the constraints that link the features and allow the configuration of tailored software products. Common SPLs involve hundreds, even thousands of features, leading to billions of possible software products. As a result, testing a product line is challenging due to the enormous size of the possible products. Existing techniques focus on testing based on the product line’s feature model by selecting a limited set of products to test. Being created manually or reverse-engineered, feature models are prone to errors impacting the generated test suites. In this paper, we examine ability of test suites to detect such errors. In particular, we propose two mutation operators to derive erroneous feature models (mutants) from an original feature model and assess the capability of the generated original test suite to kill the mutants. Experimentation on real feature models demonstrate that dissimilar tests suites have a higher mutant detection ability than similar ones, thus validating the relevance of similarity-driven product line testing.

2. Model-based verification of energy-aware real-time automotive systems

   Eun-Young Kang , Gilles Perrouin, and Pierre Yves Schobbens

   In Proceedings of the IEEE International Conference on Engineering of Complex Computer Systems, ICECCS , Oct 2013

   Abs

   EAST-ADL is an architectural description language dedicated to safety-critical automotive embedded system design with a focus on structural specification and behavioral constraints. The current concept of EAST-ADL provides limited support for modeling and analysis of Energy-aware Real-Time (ERT) behaviors due to the absence of energy constraints modeling notations and the lack of formal semantics. We address these limitations by extending the EAST-ADL notation with energy constraints and integrating this extension with formal modeling and analysis techniques. We provide a mapping scheme as the basis for automatic model transformation between the extended EAST-ADL and priced timed automata for model checking. This methodology has been implemented in a tool called A-BeTA (Aβ) and is demonstrated by means of the Brake-By-Wire case study. Our approach enables formal modeling and verification of ERT systems in EAST-ADL and identifies potential conflicts between different automotive functions at an early stage of development

3. Multi-objective test generation for software product lines

   Christopher Henard , Mike Papadakis , Gilles Perrouin, Jacques Klein , and Yves Le Traon

   In Proceedings of the 17th International Software Product Line Conference (SPLC) , Sep 2013

   Abs

   Software Products Lines (SPLs) are families of products sharing common assets representing code or functionalities of a software product. These assets are represented as features, usually organized into Feature Models (FMs) from which the user can configure software products. Generally, few features are sufficient to allow configuring millions of software products. As a result, selecting the products matching given testing objectives is a difficult problem. The testing process usually involves multiple and potentially conflicting testing objectives to fulfill, e.g. maximizing the number of optional features to test while at the same time both minimizing the number of products and minimizing the cost of testing them. However, most approaches for generating products usually target a single objective, like testing the maximum amount of feature interactions. While focusing on one objective may be sufficient in certain cases, this practice does not reflect real-life testing situations. The present paper proposes a genetic algorithm to handle multiple conflicting objectives in test generation for SPLs. Experiments conducted on FMs of different sizes demonstrate the effectiveness, feasibility and practicality of the introduced approach.

4. PLEDGE: a product line editor and test generation tool

   Christopher Henard , Mike Papadakis , Gilles Perrouin, Jacques Klein , and Yves Le Traon

   In 17th International Software Product Line Conference (co-located workshops) , Dec 2013

   Abs

   Specific requirements of clients lead to the development of variants of the same software. These variants form a Software Product Line (SPL). Ideally, testing a SPL involves testing all the software products that can be configured through the combination of features. This, however, is intractable in practice since a) large SPLs can lead to millions of possible software variants and b) the testing process is usually limited by budget and time constraints. To overcome this problem, this paper introduces PLEDGE, an open source tool that selects and prioritizes the product configurations maximizing the feature interactions covered. The uniqueness of PLEDGE is that it bypasses the computation of the feature interactions, allowing to scale to large SPLs.

5. Towards automated testing and fixing of re-engineered Feature Models

   Christopher Henard , Mike Papadakis , Gilles Perrouin, Jacques Klein , and Yves Le Traon

   In Proceedings - International Conference on Software Engineering , Oct 2013

   Abs

   Mass customization of software products requires their efficient tailoring performed through combination of features. Such features and the constraints linking them can be represented by Feature Models (FMs), allowing formal analysis, derivation of specific variants and interactive configuration. Since they are seldom present in existing systems, techniques to re-engineer FMs have been proposed. There are nevertheless error-prone and require human intervention. This paper introduces an automated search-based process to test and fix FMs so that they adequately represent actual products. Preliminary evaluation on the Linux kernel FM exhibit erroneous FM constraints and significant reduction of the inconsistencies.

6. Towards Statistical Prioritization for Software Product Lines Testing

   Xavier Devroey , Maxime Cordy , Gilles Perrouin, Pierre-Yves Schobbens , Axel Legay , and Patrick Heymans

   Oct 2013

   Abs

   Software Product Lines (SPL) are inherently difficult to test due to the combinatorial explosion of the number of products to consider. To reduce the number of products to test, sampling techniques such as combinatorial interaction testing have been proposed. They usually start from a feature model and apply a coverage criterion (e.g. pairwise feature interaction or dissimilarity) to generate tractable, fault-finding, lists of configurations to be tested. Prioritization can also be used to sort/generate such lists, optimizing coverage criteria or weights assigned to features. However, current sampling/prioritization techniques barely take product behavior into account. We explore how ideas of statistical testing, based on a usage model (a Markov chain), can be used to extract configurations of interest according to the likelihood of their executions. These executions are gathered in featured transition systems, compact representation of SPL behavior. We discuss possible scenarios and give a prioritization procedure illustrated on an example.

2012

1. A vision for behavioural model-driven validation of software product lines

   X. Devroey , M. Cordy , G. Perrouin, E.-Y. Kang , P.-Y. Schobbens , P. Heymans , A. Legay , and B. Baudry

   In Proceedings of the 5th International Symposium On Leveraging Applications of Formal Methods, Verification and Validation (ISoLA) , Oct 2012

   Abs

   The Software Product Lines (SPLs) paradigm promises faster development cycles and increased quality by systematically reusing software assets. This paradigm considers a family of systems, each of which can be obtained by a selection of features in a variability model. Though essential, providing Quality Assurance (QA) techniques for SPLs has long been perceived as a very difficult challenge due to the combinatorics induced by variability and for which very few techniques were available. Recently, important progress has been made by the model-checking and testing communities to address this QA challenge, in a very disparate way though. We present our vision for a unified framework combining model-checking and testing approaches applied to behavioural models of SPLs. Our vision relies on Featured Transition Systems (FTSs), an extension of transition systems supporting variability. This vision is also based on model-driven technologies to support practical SPL modelling and orchestrate various QA scenarios. We illustrate one of such scenarios on a vending machine SPL.

2. Behavioural Model-Driven Validation of Software Product Lines

   Xavier Devroey , Gilles Perrouin, Pierre Schobbens , Patrick Heymans , and Benoit Baudry

   Apr 2012
3. Deriving Configuration Interfaces from Feature Models : A Vision Paper

   Quentin Boucher , Gilles Perrouin, and Patrick Heymans

   In Proceedings of the Sixth International Workshop on Variability Modelling of Software-intensive Systems (VaMoS’12), Leipzig, Germany, January 25-27, , Apr 2012
4. Engineering Configuration Graphical User Interfaces: A Model-based Perspective: Paper submitted to ECMFA’12

   Quentin Boucher , Gilles Perrouin, Mathieu Acher , and Patrick Heymans

   Apr 2012
5. On Extracting Feature Models From Product Descriptions

   Mathieu Acher , Anthony Cleve , Gilles Perrouin, Patrick Heymans , Philippe Collet , Philippe Lahire , and Charles Vanbeneden

   In Proceedings of the 6th International Workshop on Variability Modelling of Software-intensive Systems (VaMoS 2012) , Apr 2012
6. Pairwise testing for software product lines: Comparison of two approaches

   Gilles Perrouin, Sebastian Oster , Sagar Sen , Jacques Klein , Benoit Baudry , and Yves Le Traon

   Software Quality Journal, Sep 2012

   Abs

   Software Product Lines (SPL) are difficult to validate due to combinatorics induced by variability, which in turn leads to combinatorial explosion of the number of derivable products. Exhaustive testing in such a large products space is hardly feasible. Hence, one possible option is to test SPLs by generating test configurations that cover all possible t feature interactions (t-wise). It dramatically reduces the number of test products while ensuring reasonable SPL coverage. In this paper, we report our experience on applying t-wise techniques for SPL with two independent toolsets developed by the authors. One focuses on generality and splits the generation problem according to strategies. The other emphasizes providing efficient generation. To evaluate the respective merits of the approaches, measures such as the number of generated test configurations and the similarity between them are provided. By applying these measures we were able to derive useful insights for pairwise and t-wise testing of product lines.

7. Simulation-Based Abstractions for Software Product-Line Model Checking

   Maxime Cordy , Andreas Classen , Gilles Perrouin, Patrick Heymans , Pierre-Yves Schobbens , and Axel Legay

   In Proceedings of the 34th International Conference on Software Engineering (ICSE 2012), Zurich, Switzerland, June 2-9 , Sep 2012
8. Towards Configurable ISO 29110-compliant Software Development Processes for Very Small Entities

   Quentin Boucher , Gilles Perrouin, Jean-Christophe Deprez , and Patrick Heymans

   In Proceedings of the 19th European System, Software & Service Process Improvement & Innovation Conference (EuroSPI’12), Vienna, Austria , Sep 2012

   Abs

   Using ISO/IEC 29110, very small entities (VSEs) can perform a step-wise increment of their software process by switching between the different ISO/IEC profiles. However, ISO/IEC 29110 provides no guidance on how to switch between profiles incrementally, other than resorting to costly software development process experts unaffordable for VSEs. To address this shortcoming, this paper shows how to model the variability of currently available ISO/IEC 29110 profiles in an integrated and configurable workflow with illustration on the Requirements Engineering (RE) activity. This workflow is linked to a questionnaire used to support automated process configuration. Thereby, the user can easily derive the ISO/IEC-compliant processes to switch between profiles incrementally. The feasibility of this approach is shown using open-source workflow management tools Synergia and YAWL.

9. Towards Flexible Evolution of Dynamically Adaptive Systems

   Gilles Perrouin, Brice Morin , Franck Chauvel , Franck Fleurey , Jacques Klein , Yves Le Traon , Olivier Barais , and Jean-Marc Jézéquel

   In Proceedings of the 34th International Conference on Software Engineering (ICSE 2012), NIER Track, Zurich, Switzerland, June 2-9 , Sep 2012
10. Towards formal energy and time aware behaviors in EAST-ADL: An MDE approach

    E.-Y. Kang , G. Perrouin, and P.-Y. Schobbens

    In Proceedings - International Conference on Quality Software , Jan 2012

    Abs

    Energy-aware real-time (ERT) systems are increasingly complex and have pervaded various areas, from automotive to telecommunication systems. Dedicated UML-based modeling languages, such as EAST-ADL or MARTE have been proposed to harness this complexity. However, they provide limited support for modeling ERT constraints, in particular continuous energy consumption. To cope with this issue we introduce a formal interchange language, eXtended Function-block Graphs (XFG), for modeling and analysis of ERT behaviors. An XFG UML profile augmenting EAST-ADL and MARTE is presented to facilitate modeling those behaviors by means of state machines. A set of mapping rules is proposed to automatically transform such profiled models into the XFG language.

11. Towards More Reliable Configurators: A Re-engineering Perspective

    Quentin Boucher , Ebrahim Khalil Abbasi , Arnaud Hubaux , Gilles Perrouin, Mathieu Acher , and Patrick Heymans

    In Proceedings of the 3rd Product LinE Approaches in Software Engineering (PLEASE’12), co-located with ICSE’12, Zurich, Switzeland , Jan 2012

    Abs

    Delivering configurable solutions, i.e. products tailored to the requirements of a particular customer, is a priority of most B2B and B2C markets. These markets now heavily rely on interactive configurators that help customers build complete and correct products. Reliability is thus a critical requirement for configurators. Yet, our experience in industry reveals that many configurators are developed in an ad hoc manner, raising correctness and maintenance issues. In this paper, we present a vision to re-engineering more reliable configurators and the challenges it poses. The first challenge is to reverse engineer from an existing configurator the variability information, including complex rules, and to consolidate it in a variability model, namely a feature model. The second challenge is to forward engineer a new configurator that uses the feature model to generate a customized graphical user interface and the underlying reasoning engine.

12. Weaving variability into domain metamodels

    Gilles Perrouin, Gilles Vanwormhoudt , Brice Morin , Philippe Lahire , Olivier Barais , and Jean-Marc Jézéquel

    Software and Systems Modeling, Jul 2012

    Abs

    Domain-specific modeling languages (DSMLs) are the essence of MDE. A DSML describes the concepts of a particular domain in a metamodel, as well as their relationships. Using a DSML, it is possible to describe a wide range of different models that often share a common base and vary on some parts. On the one hand, some current approaches tend to distinguish the variability language from the DSMLs themselves, implying greater learning curve for DSMLs stakeholders and a significant overhead in product line engineering. On the other hand, approaches integrating variability in DSMLs lack generality and tool support. We argue that aspect-oriented modeling techniques enabling flexible metamodel composition and results obtained by the software product line community to manage and resolve variability form the pillars for a solution for integrating variability into DSMLs. In this article, we consider variability as an independent and generic aspect to be woven into the DSML. In particular, we detail how variability is woven and how to perform product line derivation. We validate our approach through the weaving of variability into two different metamodels: Ecore—widely used for DSML definition—and SmartAdapters, our aspect model weaver. These results emphasize how new abilities of the language can be provided by this means.

13. XFG Language and its Profile for Modeling and Analysis of Energy-Aware Real-Time Behaviors

    Eun-Young Kang , Gilles Perrouin, and Pierre Schobbens

    Jul 2012

2011

1. A Metamodel-based Classification of Variability Modeling Approaches

   Istoan Paul , Klein Jacques , Gilles Perrouin, and Jézéquel Jean-Marc

   In VARiability for You (VARY) Workshop, co-located with MoDELS 2011 , Jul 2011

   Abs

   Software Product Line Engineering (SPLE) is an emerging paradigm taking momentum that proposes to address flexibility and shorter time-to-market by maximizing software reuse. The key characteristic of SPLE is the effective modelling and management of variability, for which a number of Variability Modeling (VM) techniques have been developed during the last two decades. Therefore, understanding their commonalities and differences is important for selecting the most suitable technique. In this paper, we propose a metamodel-based classification of VM techniques gathered through a survey of relevant literature.

2. Proceedings of the 1st International Workshop on Variability-intensive Systems Testing, Validation & Verification (VAST)

   Gilles Perrouin, Andreas Metzger , Patrick Heymans , and Yves Le Traon

   In Software Testing, Verification, and Validation Workshops (ICSTW) , Jul 2011
3. Simulation-Based Abstractions for Software Product-Line Model Checking

   Maxime Cordy , Andreas Classen , Gilles Perrouin, Patrick Heymans , Pierre-Yves Schobbens , and Axel Legay

   Jul 2011
4. VAST 2011 workshop summary

   Gilles Perrouin, Andreas Metzger , Patrick Heymans , and Yves Le Traon

   ACM SIGSOFT Software Engineering Notes, Jul 2011

2010

1. Automated and scalable t-wise test case generation strategies for software product lines

   G. Perrouin, S. Sen , J. Klein , B. Baudry , and Y. Le Traon

   In Third International Conference on Software Testing, Verification and Validation (ICST) , Jul 2010

   Abs

   Software Product Lines (SPL) are difficult to validate due to combinatorics induced by variability across their features. This leads to combinatorial explosion of the number of derivable products. Exhaustive testing in such a large space of products is infeasible. One possible option is to test SPLs by generating test cases that cover all possible T feature interactions (T-wise). T-wise dramatically reduces the number of test products while ensuring reasonable SPL coverage. However, automatic generation of test cases satisfying T-wise using SAT solvers raises two issues. The encoding of SPL models and T-wise criteria into a set of formulas acceptable by the solver and their satisfaction which fails when processed “all-at-once”’. We propose a scalable toolset using Alloy to automatically generate test cases satisfying T-wise from SPL models. We define strategies to split T-wise combinations into solvable subsets. We design and compute metrics to evaluate strategies on Aspect OPTIMA, a concrete transactional SPL.

2009

1. Composing models for detecting inconsistencies: A requirements engineering perspective

   G. Perrouin, E. Brottier , B. Baudry , and Y. Le Traon

   In Requirements Engineering: Foundation for Software Quality , Jul 2009
2. Dynamic software product lines for service-based systems

   P. Istoan , G. Nain , G. Perrouin, and J.M. Jézéquel

   In Ninth IEEE International Conference on Computer and Information Technology, CIT’09. , Jul 2009

   Abs

   Ageing populations and the necessity to reduce environmental impact raise new challenges on our living buildings. Convergence of home control systems (air conditioning, light management) and computer science, or house automation, allows to enhance comfort, security and health of inhabitants, and reduce energy consumption. Each of these abilities can be perceived as a service provided by the house automation system. Starting from this point, we developed EnTiMid , a middleware able to make systems from different brands cooperate in a single service-based platform. Yet the proliferation and variability of such services, and needs to tailor each system to a particular building, make the design of these systems complex. In this prospective paper, we explain how the notion of dynamic software product line facilitates such designs by providing sophisticated techniques for managing variability across services from design time to runtime and allowing their automatic composition.

3. SoC/SoPC development using MDD and MARTE profile

   D. Aulagnier , A. Koudri , Stéphane Lecomte , P. Soulard , J. Champeau , J. Vidal , G. Perrouin, and P. Leray

   Jul 2009
4. Weaving variability into domain metamodels

   B. Morin , G. Perrouin, P. Lahire , O. Barais , G. Vanwormhoudt , and J.M. Jézéquel

   In Model Driven Engineering Languages and Systems , Jul 2009

   Abs

   Domain-Specific Modeling Languages (DSMLs) describe the concepts of a particular domain and their relationships, in a metamodel. From a given DSML, it is possible to describe a wide range of different models. These models often share a common base and vary on some parts. Current approaches tend to distinguish the variability language from the DSMLs themselves, implying greater learning curve for DSMLs stakeholders and a significant overhead in product line engineering of DSMLs. We propose to consider variability as an independent aspect to be woven into the DSML to introduce variability capabilities. In particular we detail how variability is woven and how to perform product line derivation. We validate our approach through the weaving of variability into two very different metamodels: Ecore and SmartAdapter, our Aspect-Oriented modeling weaver, thus adding flexibility in the weaving process itself. These results emphasize how new abilities of the language can be provided by this means.

2008

1. Modeling the variability space of self-adaptive applications

   G. Perrouin, F. Chauvel , J. DeAntoni , and J.M. Jézéquel

   In 12th Software Product Line Conference , Jul 2008

   Abs

   Modeling self-adaptive applications is a difficult task due to the complex relationships they have with their environments. Designers of such applications strive to model accurately a few (re)-configuration possibilities deemed to be the most relevant with respect to environmental changes. This deliberate restriction of the variability space is cumbersome and may unnecessarily reject interesting (re)- configuration possibilities. We employ software product- line techniques to properly cover the whole variability space of a self-adaptive application. This variability space is partitioned across three dimensions. Functional variability is modeled through a feature diagram whose features are realized by a set of components to be deployed on a platform. Topological variability is modeled via an UML collaboration excluding irrelevant configurations. Platform variability is modeled through constraints to be satisfied by configurations. For each dimension, we exhibit properties capturing the environment. Our modeling approach is il- lustrated on a web-server example.

2. Reconciling automation and flexibility in product derivation

   G. Perrouin, J. Klein , N. Guelfi , and J.M. Jézéquel

   In Software Product Line Conference, 2008. SPLC’08. 12th International , Jul 2008

   Abs

   Product derivation, i.e. reusing core assets to build products, did not receive sufficient attention from the product-line community, yielding a frustrating situation. On the one hand, automated product derivation approaches are inflexible; they do not allow products meeting unforeseen, customer-specific, requirements. On the other hand, approaches that consider this issue do not provide adequate methodological guidelines nor automated support. This paper proposes an integrated product derivation approach reconciling the two views to offer both flexibility and automation. First, we perform a pre-configuration of the product by selecting desired features in a generic feature model and automatically composing their related product-line core assets. Then, we adapt the pre-configured product to its customer-specific requirements via derivation primitives combined by product engineers and controlled by constraints that flexibly set product line boundaries. Our process is supported by the Kermeta meta modeling environment and illustrated through an example.

3. Vers des Lignes de produits Flexibles

   Jean-Marc Jézéquel , and Gilles Perrouin

   L’objet, Jul 2008

2007

1. A flexible requirements analysis approach for software product lines

   N. Guelfi , and G. Perrouin

   In Proceedings of the 13th international working conference on Requirements engineering: foundation for software quality , Jul 2007

   Abs

   Product Line Engineering (PLE) promotes the development of applications by reusing a set of software assets belonging to a given domain. Important research efforts have been devoted to the description of commonalties and variabilities among these assets yielding requirements engineering techniques such as feature modeling or use case variants. However, current product derivation techniques, which strive to automate the derivation process, are inflexible in that they fail to accommodate products that represent only a minor deviation from the original product line. Furthermore, PLE methodologies do not provide precise support to assist product derivation in such cases. In this paper, we address flexibility issues by introducing an analysis model, based on UML, OCL and use cases, that implicitly defines define product line variabilities and boundaries by means of constraints forbidding undesired products. Then, in order to reuse domain assets in a coherent manner, an imperative model transformation mechanism is devised. We illustrate this approach through a simple example.

2. Architecting software systems using model transformations and architectural frameworks

   G. Perrouin

   Jul 2007

2006

1. Coherent Integration of Variability Mechanisms at the Requirements Elicitation and Analysis Levels

   N. Guelfi , and G. Perrouin

   In Variability Management – Working with Variability Mechanisms (Workshop of SPLC 06) , Jul 2006

   Abs

   Early phases of product line development can be separated in requirement elicitation and analysis. The former provides an abstract and informal description of the product line, while the latter provides a technical specification as precise and as complete as possible. The major problems we face are to define the con- tent of each phase for optimal development cycle and to provide consistency between those phases. This paper aims at integrating product line variability mechanisms between requirements elicitation and analysis levels. First, we pre- sent a requirements elicitation template based on use case variants. Then, product analysis phase is done using a generative mechanism starting from the core analysis assets and specifying the variation covered by the use case variants. These mechanisms are coherently related by means of consistency rules and the same approach is employed to integrate feature models with the analysis phase. Finally elements for variability integration reasoning are derived on the basis of these rules.

2005

1. Evolution through architectural reconciliation

   P. Avgeriou , N. Guelfi , and G. Perrouin

   Electronic Notes in Theoretical Computer Science, Jul 2005

   Abs

   One of the possible scenarios in a system evolution cycle, is to translate an emergent set of new requirements into software architecture design and subsequently to update the system implementation. In this paper, we argue that this form of forward engineering, even though addresses the new system requirements, tends to overlook the implementation constraints. An architect must also reverse-engineer the system, in order to make these constraints explicit. Thus, we propose an approach where we reconcile two architectural models, one that is forward-engineered from the requirements and another that is reverse-engineered from the implementation. The final reconciled model is optimally adapted to the emergent set of requirements and to the actual system implementation. The contribution of this paper is twofold: the application of architectural reconciliation in the context of software evolution and an approach to formalize both the specification and trans- formation of the architectural models. The architectural modeling is based upon the UML 2.0 standard, while the formalization approach is based on set theory and first-order logic.

2004

1. Using model transformation and architectural frameworks to support the software development process: The FIDJI approach

   N. Guelfi , and G. Perrouin

   In Midwest Software Engineering Conference (MSEC’04) , Jul 2004

   Abs

   Framework-based development is a well-known software engineering practice which encourages reuse and thus reduces development costs. To improve maintainability and reuse, frameworks need to be modeled quite abstractly and provide specialization mechanisms to refine these models into concrete implementation. Hence they are candidates to address Model Driven Engineering (MDE) and Model Driven Architecture (MDA) challenges. Currently, few development methodologies for MDE integrate such frameworks in their processes. In this paper, we present a development approach called FIDJI, which bases its process on architectural framework specialization using model transformation and its associated CASE tool support.

2003

1. Supporting model-to-model transformations: The vmt approach

   S. Sendall , G. Perrouin, N. Guelfi , and O. Biberstein

   In Model Driven Architecture Foundations and Applications , Jul 2003

   Abs

   The model-driven architecture approach (MDA) promotes software development as driven by a thorough modeling phase where application code is automatically generated from a platform specific UML model (PSM). The idea is that the PSM is itself derived from a platform independent UML model (PIM). Such code generation and model derivation are examples of model-to-model transformations.

2002

1. Rigorous engineering of software architectures: Integrating ADLs, UML and development methodologies

   N. Guelfi , and G. Perrouin

   In Software Engineering and Applications , Jul 2002

   Abs

   Architecture engineering is an area that is under construction for being transferred to software engineers. Many results exist from high-level formal and conceptual approaches to basic good practices taken from industrial experiences. Descriptions and methodologies should be at least considered for architecture engineering. Two interesting approaches to architecture description are avail able: Architecture Description Languages (ADL) or modeling languages (especially UML). In the first case, efforts have been made to provide rigorous integrated languages dedicated to architecture engineering but unfortunately these languages are incomplete and not ready for technology transfer. Concerning UML, many approaches use UML models for partial architecture descriptions. In both cases, a few efforts have been made on methodology. This paper presents the current state of the art covering architecture descriptions and development methodologies for architecture engineering. We present a first integrated approach, called FIDJI that tries to take advantage from ADLs, UML, and some development methodologies. FIDJI uses UML for its models, MDA for its layering and ADLs for precision. Concerning methodology, FIDJI follows the HP recommendations and the notion of refinement and contracts taken from formal development methodologies. Finally, we briefly illustrate the use of the FIDJI approach by means of a real web-based distributed application.