Logs and models in engineering complex embedded production software systems

Embedded production systems are nowadays widely used in production and manufacturing for a more precise control of the production processes and the quality of products. Such systems are hard to maintain due to their complex nature. They consist of thousands of software and hardware components that are communicating with each other, capture the concepts and designs related to different disciplines (e.g., physics and chemistry), and are often featured with critical performance requirements. The portion of system functionality realized by software is increasing in such systems, rendering it increasingly complex. According to the existing empirical studies, the general software engineering methodologies and techniques might not be sufficient to address the challenges raised by the properties of embedded production software, suggesting the need for proposing tailored techniques for such systems. As the first step towards proposing effective techniques, it is essential to understand the current practices and the challenges developers are facing. To gain this understanding, we conduct a series of empirical studies at ASML, a leading manufacturer of lithography machines for semiconductor industry.

Similar to many other complex systems, systems at ASML generate a large number of logs that capture the runtime behavior of the systems. Due to the presence of rich information, execution logs are considered to be essential inputs for software analytics tools and processes that aim at addressing the complexity of large-scale systems. To provide useful and effective software analytics tools for complex embedded systems, it is hence necessary to understand how the developers of such systems use logs in practice. We conduct an interview study with 39 software developers. We first perform a series of interviews with 25 developers at ASML and then replicate the interviews at four other companies with another 14 software developers. In this interview study, we learn that developers often compare logs generated from multiple executions to support their maintenance activities such as root cause analysis and behavioral verification. While many log comparison techniques have been proposed in the academic literature, text-based editors are the commonly used tools for this practice. This observation leads us to study the existing log comparison techniques and their limitations by conducting a literature review about the existing log comparison techniques. This literature study reveals that most of the existing log comparison techniques do not explicitly take the industrial challenges into account and were evaluated in a limited way without involvement of human participants. To provide software developers with effective log comparison techniques, we suggest researchers to improve log comparison techniques to address the industrial challenges and evaluate the techniques in a natural development setting.

In the interview study about log analysis practice, we also observe that developers often manually sketch behavioral models based on logs. This preference of presenting log information with models co-occurs with the transition ASML is taking from code-based software engineering to model-driven software engineering (MDSE). To enable the use of MDSE, ASML needs to create models for the existing code-based components. To facilitate the automation of model creation, we propose a model inference technique that can extract models by combining log analysis, and analysis of a running system under stimuli. The proposed technique significantly outperforms the existing techniques, as evaluated with 18 ASML software components. However, there are still many theoretical and practical challenges to be addressed in order to apply model inference techniques in industry. Due to these challenges, models are manually created by software developers in practice. We therefore turn our attention to study how developers manually create models for MDSE. Particularly, we study why developers violate modeling guidelines which are considered as common wisdom. We focus on an extreme case, known as flower models, consisting of only a single state. Combining qualitative and quantitative analyses, we identify the main reasons of the guideline violations, providing empirical evidence on the challenges in MDSE, and suggestions on the improvements of MDSE tools and guidelines.

In summary, this thesis presents a series of empirical studies conducted in industry, which provides an overview of challenges faced by developers when using logs and models in the context of a transition from code-based engineering to MDSE. The empirical evidence collected from these studies supports researchers and tool builders to develop techniques for facilitating the transition to MDSE for embedded production systems.