First Year of ECOLE – A Review

Time flies fast. It has been almost over one year since I joined ECOLE. And it has been an exciting year. Travelling between Birmingham, Warsaw, Leiden, Berlin, Coimbra and Xiamen. Listening to talks in state of the art on research in Evolutionary Computation and Machine Learning. Attending workshops in the research institutions of two world-known tech companies. The opportunities promised and given by the Innovative Training Network have not fallen short. But aside from all the excitement and activities, who am I, what do I do and what is that drove me to ECOLE in the first place?

Well, my prior academic pursuits were in Theoretical Nuclear Physics at the Technical University Darmstadt. And while I of course enjoyed studying the field and gaining understanding into the inner subnuclear workings of the world, I felt at times the research self was missing out in some way on my initial fascinations which took me too it. So, where to go else? Stumbling upon ECOLE was to be honest a bit of a lucky coincidence. I was working prior at software company while looking for PhD and residency programs, when a friend of mine was recommending me the posting for an Early Stage Researcher position he found at one of the partner institutions. I applied, was interviewed and subsequentially have been offered the position. It has been a bit unusual for me to travel this much for the Marie Curie Fellowship, but I am happy to have found a new home. And with Computational Intelligence research being situated at the intersection between Natural Computation and Artificial Intelligence, there is still plenty of space for me to philosophize about nature, while keeping in touch with reality through real-world problems.

ECOLE is especially great in regards of opportunities to learn, grow and try out new things. I particularly had a lot of fun (but also stress) with preparing and giving a short keynote-style public talk on nature-inspired artificial intelligence for the popular science segment of the Z2X festival in Berlin last August. The talk was luckily to be held lightweight and in front of a small audience. I decided to talk about the history and inspirations behind current popular methods and highlight evolutionary approaches as a means for creative problem-solving. While part of the audience seemed to have got lost with some of the bare technicalities, the main ideas sticked. Which was the most important thing for me. I was thus more than happy to have received and interesting questions in return. By the way: I am planning to put my practice slides online soon, if you are interested, feel free to take a look. The actual talk was given completely in oral form, as I was informed one day prior that the beamer was missing, but it worked out.

So, what’s up for the future? Currently, I work on model-based approaches as a means to tackle the transfer learning problem in continuous optimisation. A bit of a complicated idea, but I find this notion to be intuive and interesting to explore. In any way, I will keep you updated about my work in a future blog post. Until then, everyone keep up with the good work! 🙂

MIP-EGO4ML: A python Hyperparameter optimisation library for machine learning

Introduction

Machine learning techniques become essential for the research on big data as well as applications on real-world problems. For instance, many machine learning algorithms are effective in areas, e.g., handwriting recognition, time series forecasting, self-driving etc. Despite its high applicability, most of the machine learning system has various hyperparameters, which affect the performance of those algorithms significantly. Thus, it is crucial to investigate how good configurations of hyperparameters can be obtained in an automatic manner for real-world applications. This consideration naturally leads to the well-established Automatic Configuration methodology.

Hyperparameter optimisation

There are two common techniques for hyperparameter optimisation are Grid-search and Random-search [2, 3]. Grid-search is a traditional method, which will be studying every point in the search space. However, it considerably expensive due to the number of search points needed to achieve. Instead of searching on full search points, Random-search will choose random points, that will reduce the studying cost. However, the effect of this approach is completely random. Bayesian optimisation [2, 3, 4] is more modern techniques, which aims to predict and suggest new promising search points for the optimizer. In this blog post, we will focus on one of our implementation of Bayesian optimization, a Python hyperparameter optimisation library called MIP-EGO4ML.

MIP-EGO4ML: A python hyperparameter optimisation library for machine learning

The Hyperparameter Optimisation (HPO) library “MIP-EGO4ML” is a python library based on the MIP-EGO library (available at http://hyperparameter.ml). In principle, MIP-EGO4ML is an extended version of the MIP-EGO library with conditional search space added. Thus, if your search space does not contain any conditional parameters, that means the MIP-EGO library is processing your HPO process.

MIP-EGO4ML is very simple and easy to use. To run the library, we only need three steps:

1. Objective function: the function must return a real value.

2. Configuration space: hyperparameters for the objective function

3. Configuration of MIP-EGO4ML.

Firstly, we must have an objective function, which must return a single real-value number to minimize or maximize. In this article, we will use a simple classification problem that will describe how to run an optimize for a classification problem by MIP-EGO4ML.

And then, we need define a Configuration space or hyperparameter space for this objective function, which we want to optimize.

At the final step, we need to set the configuration for MIP-EGO4ML.

The code for this article is available in a Jupyter Notebook on GitHub.

This article only provides the basic introduction about our novel hyperparameter optimisation library; we will update more article about hyperparameter optimisation techniques in the future posts.

References

[1]. Gelzinis, A. Verikas, E. Vaiciukynas, M. Bacauskiene, J. Minelga, M. Hållander, V. Uloza and E. Padervinskis, “Exploring sustained phonation recorded with acoustic and contact microphones to screen for laryngeal disorders,” in Conference: 2014 IEEE Symposium on Computational Intelligence in Healthcare and e-health (CICARE), 2014.

[2]. M. Feurer and F. Hutter, “Hyperparameter Optimization,” in AutoML: Methods, Sytems, Challenges, Springer, 2018, pp. 3-37.

[3]. H. H. Hoos, “Automated Algorithm Configuration and Parameter Tuning,” in Hamadi Y., Monfroy E., Saubion F. (eds) Autonomous Search, Berlin, Heidelberg, Springer, 2011, pp. 37-71.

[4]. H. Wang, B. v. Stein, M. Emmerich and T. Bäck, “A new acquisition function for Bayesian optimization based on the moment-generating function,” 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 507-512, 2017.

Experience of My First Paper and Conference

Experience of Writing My First Conference Paper:

It is now more than a year since I started working for the ECOLE project. It has completely changed my concept of pursuing a Ph. D, i.e., from understanding the perspective of subjects of industrial use cases to approach a technical problem in gradual steps. Initially, I had spent a lot of time in trying to understand the problem by studying related literature and research papers while performing initial experiments. After nine months since the start of the project, I decided to write my first conference paper. The most difficult part that I faced while writing my first paper was structuring the paper. Initially, my focus was on carrying out a number of experiments, gathering results and plotting graphs instead of starting to write the paper in parallel.  If I look back and think about my experience, I realize I had made several mistakes. These include the delay in starting to write the manuscript and circulating the draft several times. In the end, I was confused with the large number of items to include in a single paper and in which section to insert each one of them. Also, it was a collaborative paper with another researcher. It gave me the pleasure of working together for a project. Still in the end, we ended up working till the last moment on the last date of the submission. Fortunately, our work paid off and the paper was accepted and I got the chance to attend my first conference.

Experience of Attending the First Conference:

It was my first experience of attending a conference and also my first visit to China.  We submitted our paper in the Learning Mining with Industrial Data (LMID 2019) held in conjunction with the International Conference on Data Mining (2019 IEEE) from 8^th November to 11^th November in Beijing, China.  The workshop was related to machine learning and data mining of industrial data. On the first day of the conference, there were talks by Prof Bodgan Gabyrs on automation as a predictive system and from Prof Xin Yao on ensemble approaches to class imbalance problems. There were eight other interesting presentations from different researchers. There were also several other keynote talks on the application of AI related to Industrial applications like in transportation, social networking, medical diagnosis, etc.  Other than attending this workshop, there were a number of keynote speakers and tutorials on the four days’ long conference. Listening to all these talks has actually helped me to think out of the box of daily research. Also, it gave me an opportunity to meet other researchers, to learn networking and enjoy some really good food.  Personally, I enjoyed the four-day long conference in Beijing.

Short Introduction about the Class Imbalance Classification Problem

What is a class imbalance classification problem?

Strictly speaking, any dataset that contains an unequal class distribution between its classes can be considered as imbalanced [1]. However, in the community, only the datasets that contain significant or extreme imbalance are regarded as imbalances datasets. An illustration of class imbalance problem is shown in Figure 1. Even if the classifier predicts all the samples as majority class, the accuracy is still 95%, which makes the classifier seems extremely efficient but actually it neglects the minority class.

Several steps to deal with class imbalance classification problem.

From Figure 1, we know that the accuracy does not reflect the actual effectiveness of an algorithm in imbalanced domains. Hence, the first thing needs to be done to deal with imbalanced dataset is to change the performance evaluation metrics. The Area Under the ROC curve (AUC), F-measure, geometric mean are three commonly used performance metrics (detailed information for the metrics can be found in [1]). Over years of development, many techniques have proven to be efficient in handling imbalanced datasets. These methods can be divided into data-level approaches and algorithmic-level approaches [2, 3], where the data-level approaches aim to produce balanced datasets and the algorithmic-level approaches aim to adjust classical classification algorithms in order to make them appropriate for handling imbalanced datasets.

Two popular data-level approaches.

Data-level approaches, as known as resampling techniques, can be divided into undersampling (under sample majority class samples) and oversampling techniques (oversample the minority class samples). Here we only introduce two popular oversampling techniques.

The synthetic minority over-sampling technique (SMOTE), proposed in 2002, is the most popular resampling technique [4]. SMOTE produces balanced data through creating artificial data based on the randomly chosen minority samples and their K-nearest neighbors. The procedure to generate a new synthetic sample is shown in Figure 2.

The adaptive synthetic (ADASYN) sampling technique is a method that aims to adaptively generate minority samples according to their distributions [5]. The main improvement compared to SMOTE is the samples which are harder to learn are given higher importance and will be oversampled more often in ADASYN. The general idea of ADASYN is shown in Figure 3.

We only include the most basic knowledge of class imbalance in this blog and more tutorial about class imbalance will be posted later.

References

[1]. He, H. and Garcia, E.A., 2009. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9), pp.1263-1284
[2]. Ganganwar, V., 2012. An overview of classification algorithms for imbalanced datasets. International Journal of Emerging Technology and Advanced Engineering, 2(4), pp.42-47.
[3]. Santos, M.S., Soares, J.P., Abreu, P.H., Araujo, H. and Santos, J., 2018. Cross-validation for imbalanced datasets: Avoiding overoptimistic and overfitting approaches [research frontier]. ieee ComputatioNal iNtelligeNCe magaziNe, 13(4), pp.59-76.
[4]. Chawla, N.V., Bowyer, K.W., Hall, L.O. and Kegelmeyer, W.P., 2002. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16, pp.321-357.
[5]. He, H., Bai, Y., Garcia, E.A. and Li, S., 2008, June. ADASYN: Adaptive synthetic sampling approach for imbalanced learning. In 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence) (pp. 1322-1328). IEEE.

Crucial Facets of Modern Time Series Analysis

Time series analysis is an important motif in the domains of “Macroeconomics”, “Financial Portfolio Management”, “Weather Forecasting”, “Disaster Management” and “Health Care” to name a few. With the age of digitalization, the industrial and ubiquitous systems such as smart phones, health care units and economic transaction processing systems produce huge volume of time ordered data. Understanding the complex and dynamic processes that produce these time ordered data can lead to significant benefits for the society and businesses. This require a holistic analysis of the time series generated by these processes including feature selection, model selection, validation and prediction of future time related events. In this blog post, I try to summarize some of the major challenges in time series analysis that require the attention of scientific community. Two example signals, respectively in Figure 1 and 2 are plotted for this blog. Figure 1 shows the plot of daily minimum temperature in Celsius as measured by the Australian bureau of Meteorology for the city of Melbourne from 1981 to 1991. Figure 2 shows the plots concerning the power consumption of all households in Germany, and the solar and wind power generation in the country from 2006-2018. The unit of measurement for Fig 2 is Giga watthours (GWh).

The classical analysis of time series relies heavily on “Auto Regressive (AR)” and “Exponential Smoothing” models [1] which are typically linear and are suited for univariate time series. These state space methods involve human expertise in the loop and require explicit specifications of trend, seasonality, cyclical effects and shocks when modeling time series. As a result, these methods are interpretable and the prediction is ready to use. However, this interpretability comes at the sacrifice of prediction accuracy since such methods lack the dynamic and complex nature of modern industrial and ubiquitous processes with multivariate time series e.g., stock changes, energy consumption in power plants etc. Recently, with the advent of empirical models such as deep neural networks [2], there has been a growing interest in time series modeling using deep learning. This usually involves modeling time series by a sequential neural network architecture such as Recurrent Neural Network and its variants. The predictions for future observations can be made once the parameters of the neural network are optimized using any methodology to compute gradients in recurrent structures e.g., Back Propagation Through Time (BPTT), Real Time Recurrent Learning (RTRL) [3] etc. Although the neural networks enjoy higher accuracy than linear state space models in the context of time series, their prediction is often not interpretable since they behave as black box models. Additionally, the computational complexity and choice of architecture make things much more complicated. Hence, we’re left with the dilemma of simplicity vs complexity, both of which have their own pros and cons.

Recently, hybrid models such as deep state space models [4] have been proposed in the context of time series analysis which combine the best of both worlds to produce robust predictions. Hybrid models have the advantage of working with both simple and complex time series. This further means that size of time series does not affect the learning itself since for small data, state space models make full use of the structural assumptions in the data while for large data sets, deep neural networks can capture the complex and nonlinear relationship and hence, the resulting model usually perform better than a stand-alone model. It is important to understand that despite the advent in hybrid models, there’re still major issues in time series analysis that need the attention of scientific community. This involve feature selection, model selection, cross validation and interpretation in time series analysis. To this end, Bayesian time series frameworks [5] have been proposed that take care of feature and model selection however such frameworks have the drawback of computational complexity since estimating the posterior using MCMC models is computationally expensive for higher dimensions.

Added to these are the issues of missing data imputation, irregular & asynchronous time series [6], non-causal and non-invertible processes, categorical/qualitative processes, class imbalance learning, and mechanisms for early outliers and changepoints detection [7] i.e., structural breaks. Not addressing these issues may have grave consequences in the context of learning and the predictions might be misleading. Finally, there’s lack of cohesive literature in Multivariate Time Series Analysis and consequently there is no consensus on the selection of multiple related time series based on the information from the application domain e.g., disease diagnostics in the case of electronic health records in ICU etc. which could be useful for inference.

References

[1] Lütkepohl, Helmut. New introduction to multiple time series analysis. Springer Science & Business Media, 2005.
[2] LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. “Deep learning.” nature 521.7553 (2015): 436-444.
[3] Williams, Ronald J., and David Zipser. “A learning algorithm for continually running fully recurrent neural networks.” Neural computation 1.2 (1989): 270-280.
[4] Rangapuram, Syama Sundar, et al. “Deep state space models for time series forecasting.” Advances in Neural Information Processing Systems. 2018.
[5] Qiu, Jinwen, S. Rao Jammalamadaka, and Ning Ning. “Multivariate Bayesian Structural Time Series Model.” Journal of Machine Learning Research 19.68 (2018): 1-33.
[6] Harutyunyan, Hrayr, et al. “Multitask learning and benchmarking with clinical time series data.” arXiv preprint arXiv:1703.07771 (2017).
[7] Guo, Tian, et al. “Robust online time series prediction with recurrent neural networks.” 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA). Ieee, 2016.

Sharing Experience of First Year in ECOLE

Personal Background:

Not much longer than one year ago, at the end of my Master’s degree and after 8 years in the Mechanical Engineering department at UFSC (Brazil), starting a PhD was not exactly what I had in mind. I have seen many friends in the post-graduate program doing pure academic research, sometimes without perspective of any real-world industrial application for their developments. Since I experienced several extra-curricular activities in parallel to my studies – 3 one-year internships and 6 years dedicated to projects in student racing teams – I did not see myself working on an exclusively academic environment and enduring for additional three or four years of lectures and publications, even though the doctorate program there was really good. On top of that, I was afraid that what I learned during practical assignments would slowly vanish away in order to leave space for new, but highly specialized, knowledge.

Motivation to apply for ECOLE:

The proposal of the ECOLE project and the concept of the Innovative Training Networks (ITN) challenged me with the idea that pursuing the PhD degree could be very different to what I expected. Spending three years among industry and academia; living in two different countries; and being able to finish a dissertation, papers and training modules sounded very intense even before the job interview. However, the aspect of bringing the fundamental research together to real-world industrial applications was exactly what I was looking for; and the particularity of the ECOLE being applied to the automotive engineering, which is deeply related to my educational background, really pushed me to join the team.

Experience in ECOLE:

Now, almost one year after my start in the project, it is satisfying to realize how much this interesting mix between academia and industry, computer science and engineering, improved my technical and professional skills. The difficulties to fit the research into engineering applications still exist – several years of practice with mechanical engineering tasks help me tough – but it has become easier to face them as part of the research and take it as a motivation to move forward with the PhD. The support of the supervisors has been essential: through constructive feedbacks and fruitful scientific discussions, we already achieved great results (nine accepted publications distributed among the ESRs in the project!) and put great ideas together for future research.