Learning in Abstract Memory Schemes for Dynamic Optimization

Transcription

1 Fourth International Conference on Natural Computation Learning in Abstract Memory Schemes for Dynamic Optimization Hendrik Richter HTWK Leipzig, Fachbereich Elektrotechnik und Informationstechnik, Institut Mess, Steuerungs und Regelungstechnik, D 05 Leipzig, Germany Shengxiang Yang Department of Computer Science University of Leicester, University Road, Leicester LE 7RH, United Kingdom Abstract We investigate an abstraction based memory scheme for evolutionary algorithms in dynamic environments. In this scheme, the abstraction of good solutions (i.e., their approximate location in the search space) is stored in the memory instead of good solutions themselves and is employed to improve future problem solving. In particular, this paper shows how learning takes place in the abstract memory scheme and how the performance in problem solving changes over time for different kinds of dynamics in the fitness landscape. The experiments show that the abstract memory enables learning processes and efficiently improves the performance of evolutionary algorithms in dynamic environments. Introduction A main concern of evolutionary algorithms (EAs) for dynamic optimization problems (DOPs) is to maintain the genetic diversity of the population [5, 7]. Only this will guarantee continuing and sustainable evolutionary search for optima that change with time. To achieve the maintenance of diversity, two main concepts have been applied. One is to preserve diversity by mainly random means, which is realized by designs such as hyper mutation [9] and random immigrants []. Another is to promote diversity by basically deterministic methods through saving individuals or groups of individuals for future reinsertion or merger. Such ideas are implemented in memory [, 8] and multi population approaches []. Although both concepts have shown to be successful for certain dynamic environments, there are some points of criticism. One is that they do not or do not explicitly incorporate information about the dynamics and hence do not discriminate between different kinds of dynamic fitness landscapes. A second concern is the usage of past and present solutions for improving the quality of future solution finding. This aspect is not addressed by random diversity enhancement. In contrast, memory techniques do use previous good solutions. This is done in implicit memory schemes by redundant representation using multiploidy and dominance [8]. Explicit memory stores good solutions (sometimes accompanied by information about environmental conditions) for later reuse [, ]. Here, it is natural to ask how and why this brings improvements in performance and an obvious answer is that by storing and reusing information some kinds of learning processes are carried out. However, the detailed relationships between memory and learning in dynamic optimization are poorly studied. Memory schemes that only store good solutions as themselves, known as direct memory [], for later reinsertion carry out learning processes implicitly at best. Learning is something different than memorizing all previous solutions. In parts, this might be helpful. In general, every realizable memory will soon prove insufficient in a more complex context; if not by storing capacity itself, then by a timely retrieval of the stored content for further usage. In the more wider sense discussed above, learning refers to detecting the essence and meaning of a solution. The abstract memory scheme proposed here, which is detailed in Sec., intends to address and employ these relations. Abstraction means to select, evaluate and code information before storing. A good solution is evaluated with respect to physically meaningful criteria and in the result of this evaluation, storage is undertaken but no longer as the solution itself but coded with respect to the criteria. So, abstraction means a threshold for and compression of information, see e.g. [6] which proposes similar ideas for reinforcement learning. So, the /08 $ IEEE DOI 0.09/ICNC

2 scheme we present is not merely concerned with anticipating the dynamics of the fitness function alone, as considered in [], but to predict where good solutions of the dynamic optimization problem are likely to occur. Hence, we bring together learning and memory for evolutionary optimization in dynamic environments. This paper presents an abstraction based memory scheme for dynamic optimization. We show that such a memory scheme enables learning processes conceptionally and functionally similar to those considered in machine learning. It explicitly uses past and present solutions in an abstraction process that is employed to improve future problem solving and differentiates between different kinds of dynamics of the fitness landscape. In particular, we intend to study how learning takes place in the memory scheme. Therefore, we consider how the performance in problem solving changes over time for different kinds of dynamics in the fitness landscape, which are regular, chaotic, and random. The rest of this paper is outlined as below. Sec. states the DOP and the EA to solve it. The abstract memory scheme is given in Sec.. Experiments are reported and discussed in Sec.. Sec. 5 concludes the paper with discussions on future work. Solving the DOP Learning behavior in abstract memory schemes occurs in solving a DOP, which in this study is posed as follows. We employ as dynamic fitness landscape an n dimensional field of cones on a zero plane, where N cones with coordinates c i (k), i =,,..., N are moving with discrete time k N 0. These cones are distributed across the landscape and have randomly chosen initial coordinates c(0), heights h i and slopes s i. So, we will employ the dynamic fitness function { f(x, k) = max 0, max [h i s i x c i (k) ] }. i N () The EA we use has a real number representation and individuals x j R n, j =,,...,, which build the population P R n. Its dynamics is described by the generation transition function ψ : R n R n, see e.g. [], p It can be interpreted as a nonlinear probabilistic dynamical system that maps P (t) onto P (t +)by the standard genetic operators selection, recombination and mutation as well as by using an abstract memory, which is described in detail in Sec.. It hence transforms a population at generation t N 0 into a population at generation t +, P (t +) =ψ (P (t)),t 0. Starting from an initial population P (0), the population sequence P (0), P (), P (),... describes the temporal movement of the population in the search space. Both the time scales t and k are related by the change frequency N as t = k. () For =, apparently, the dynamic fitness function is changing every generation. For >, the fitness function changes every generations. The change frequency is an important quantity in dynamic optimization and will be the subject of the experimental studies reported in Sec.. The Abstract Memory Scheme The main idea of the abstract memory scheme is that it does not store good solutions as themselves but as their abstraction. We define an abstraction of a good solution to be its approximate location in the search space. Hence, we need to partition the search space. This can be obtained by partitioning the relevant (bounded) search space into rectangular (hyper ) cells. Every cell can be addressed by an element of a matrix. So, for an n dimensional search space M we obtain an n dimensional matrix whose elements represent search space sub spaces. This matrix acts as our abstract memory and will be called memory matrix M. Itis meant to represent the spatial distribution of good solutions. The abstract storage process consists of two steps, a selecting process and a memorizing process. The selecting process picks good individuals from the population P (t) while the EA runs. In general, selection has to be done in terms of (i.) the amount and choice of considered individuals (ideally sorted according to their fitness) from the population and (ii.) points in the run time of the EA (ideally sorted according to changes in the environment). For the individuals, either only the best or a few best from the population could be used. In terms of the run time between changes only the best over run time or the best over a few generations before a change occurs could be taken. We define the number of the individuals selected for memorizing as well as the number of generations where memorizing is carried out. In the memorizing process, the selected individuals are sorted according to their partition in the search space which they represent. In order to obtain this partition, we assume that the search space M is bounded and in every direction there are lower and upper bounds, x imin and x imax, i =,,...,n. With the grid size ɛ, we obtain for every generation t the memory matrix M(t) R h h... hn, ximax xi min where h i = ɛ. In the memory M(t) the entry of each element m ll...l n (t) is a counter count ll...l n (t), l i =,,...,h i, which is empty for initialization, that is, count ll...l n (0) = 0 for all l i. For each individual x j (t) P (t) selected to take part in the memorizing, the counter of the element representing the partition that the individual belongs to is increased by one. That is, we calculate the index l i = xij xi min ɛ for all x j =(x j,x j,...,x nj ) T 87

3 (a) chaotic (b) normal (c) uniform (d) cyclic Figure. Performance of the EA measured by the over change frequency for different types of dynamics and no memory but hypermutation (), direct memory () and abstract memory (). and all i n and increment the corresponding count l l...l n (t). Note that this process might be carried out several times in a generation t if more than one individual selected belongs to the same partition. The abstraction storage process retains the abstraction of good solutions by accumulating locations where the good solution occurred. In this way, we encode and compress the information about good solutions. As the matrix M is filled over run time, the memorizing incorporates a learning process. After a change has been detected (usually if the sliding mean of the best individual is falling), the abstract retrieval process is carried out. It consists of two steps. First, a matrix M µ (t) is calculated by dividing the matrix M(t) by the sum of all matrix elements, that is M µ (t) = h M(t) M(t). Hence, the sum i of all elements µ l l...l n (t) in M µ (t) adds up to one. Each element in M µ (t) contains an approximation of the natural measure µ li [0, ] belonging to the corresponding partition cell M l l...l n of the search space M. This natural measure can be viewed as the probability of the occurrence of a good solution within the partition over time of the dynamic environment. Hence, it is a probabilistic mapping between search space cells and the expected value for a good solution within each cell. Next, we fix the number of individuals to be created by τ, τ and create these individuals randomly such that their statistical distribution regarding the partition matches that stored in the memory M µ (t). Therefore, we first determine the number of individuals to be created for each cell by sorting the µ l l...l n (t) according to their magnitude and set the number µ l l...l n (t) τ of new individuals for high values of µ and the number µ l l...l n (t) τ for low values, respectively. The rounding needs to ensure that µl l...l n (t) τ + µ l l...l n (t) τ = τ. Then, we fix the positions of the new individuals by taking realizations of a random variable uniformly distributed within each partition cell M l l...l n. This means the τ individuals are distributed such that the number within each cell approximates the probability of the occurrence of good solutions, while the exact position within partition cells is random. These individuals are inserted in the population P (t) after mutation has been carried out. This abstract retrieval process can create an arbitrary number of individuals from the abstract 88

4 (a) chaotic (b) normal (c) uniform (d) cyclic 00 Figure. The depending on the population size and the number of individuals retrieved from the memory τ, given as percentage τ/ in %. memory. In the implementation considered here we upper bound this creation by the number of individuals in the population. As abstract storage can be regarded as encoding and compression of information about good solutions in search space, abstract retrieval becomes decoding and expansion. Numerical Experiments The numerical results given here are obtained with an EA that uses tournament selection of tournament size, fitness related intermediate sexual recombination, a mutation operator with base mutation rate 0. and the proposed abstraction memory scheme. We measure performance by the Mean Fitness Error = [ R ] T R r= T t= (f best(k) f max (k)) where k= t f best (k) = max f ( x j (t), t ) is the fitness value x j(t) P (t) of the best in generation individual x j (t) P (t) at generation t, f max (k) = f ( x s ( t ), t ) is the maximum fitness value at generation t, T is the number of generations used in the run, and R is the number of consecutive runs. Note that f ( x s, t ) and max f ( x j (t), t ) change every generations according to Eq. (). x j(t) P (t) The parameters in all experiments are R = 50 and T = 000. We consider the dynamic fitness function () with dimension n =and the number of cones N =7.We study four types of dynamics of the coordinates c(k) of the cones; (i.) chaotic dynamics generated by the Hénon map, see [0] for details of the generation process, (ii.) random dynamics with c(k) being realizations of a normally distributed random variable, (iii.) random dynamics with c(k) being realizations of a uniformly distributed random variable, and (iv.) cyclic dynamics where c(k) are consequently forming a circle. As the dynamic severity is an important factor in dynamic optimization, severity is normalized for all considered dynamics and hence has no differentiating influence. In a first set of experiments, the abstract memory scheme () is tested and compared with a direct memory scheme (), which stores the good solutions and inserts them again in a retrieval process, and with an EA with no memory (), that uses hypermutation with the hypermutation rate set to 0, see Figure. Here, as well as in the 89

5 5 5.5 = 5.5 = 5 = 5 = 5 = 5 = L L (a) chaotic (b) normal.5 = 5.5 = 5 = 5 = 5 = 5 = L L (c) uniform (d) cyclic Figure. Learning curves for the abstract memory scheme showing the learning success measured by L over learning time t L. other experiments we fixed the upper and lower bounds of the search space at x min = x min = and at x max = x max =. The best three individuals of the population take part in the memorizing process for all three generations before a change in the environment occurs. Further,wesetthegridsizetoɛ =0.. We used a fixed population size of =50and inserted τ =0individuals in the retrieval process for one generation after the change. In Figure we give the over the change frequency for all four types of dynamics considered. Also, the 95% confidence intervals are given. We observe that the memory schemes outperform the no memory scheme for all dynamics. This is particularly noticeable for small change frequencies and means that by memory the limit of for which the algorithm still performs reasonably can be considerably lowered. Also, it can be seen that the abstract memory gives better results than the direct memory for irregular dynamics, that is, chaotic and random. For chaotic dynamics, this is even significant within the given bounds. For regular, cyclic dynamics we find the contrary, with direct memory being better than abstract. A second set of experiments examines the relationship between the populations size, the number of retrieved individuals τ and the performance measure, see Figure, which shows results for the fixed change frequency =5. We see an exponential relationship between and, which is typical for EAs. Along this general trend, the number of retrieved individuals, here given in percent of the total population, has only a small influence on the, where in general a medium and large number gives slightly better results than a very small percentage. To quantify learning depends on metrics for performance, which ideally shows improvement over time. For evaluating the effect of learning and obtaining the learning curve, the experiment has to enable learning for a certain time, then turn learning off and measure the performance using the learned ability. Regarding the abstract memory scheme considered here, learning takes place as long as the memory matrix M µ (t) is filled. This gives raise to the following measure for learning success. We define t L to be the learning time. For 0 <t t L the matrix M µ (t) is filled as described in Sec.. For t L <t<tthe storage process is discarded and only retrieval from the now fixed memory is carried out. We calculate the for 90

6 t>t L only and denote it L. It is a performance measure for the learning success, where L over t L shows the learning curve. Figure depicts the results for fixed =50, τ =0and several change frequencies on the semi logarithmic scale. These learning curves are an experimental evaluation of the learning behavior. We see that the L gets gradually smaller with the learning time t L becoming larger, which confirms the learning success. We find a negative linear relation between and log(t L ), which indicates an exponential dependency between and t L. Also, it can be seen that the learning curves are slightly steeper for larger change frequencies. An exception to this general trend is cyclic dynamics, where the learning curves are almost parallel for all and a large proportion of the tested t L. A comparison of the learning success between the different kinds of landscape dynamics suggests that the uniform random movement is the most difficult to learn. To summarize, the results in Figure clearly indicate the positive effect of learning on the performance of the EA. 5 Conclusions This paper investigates an abstract memory scheme for evolutionary algorithms in dynamic environments, where memory is used to store the abstraction of good solutions (i.e., their approximate location in the search space) instead of good solutions themselves. This abstraction is employed to generate solutions to improve future problem solving. In order to understand the relationships between memory and learning in dynamic environments, experiments were carried out to investigate how learning takes place in the abstract memory scheme and how the performance changes over time for different kinds of dynamics in the fitness landscape. The experimental study reveals several results on the dynamic test environments. First, it is usually beneficial to integrate memory schemes into EAs for DOPs and the abstract memory performs better than direct memory in chaotic and random dynamics. Second, the abstraction based memory scheme enables learning processes, which efficiently improves the performance of EAs in dynamic environments. Third, the effect of the abstract memory on the performance of the EA depends on the learning time and the frequency of environmental changes. This paper studied the relations between learning and the abstract memory in dynamic environments. For the future work, it is valuable to further investigate this important but poorly studied issue, the relations between learning and other memory schemes for DOPs. Another interesting research is to compare and combine the abstract memory scheme with other approaches developed for EAs in dynamic environments. References [] T. Bäck. Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms, Oxford University Press, NY, 996. [] P. A. N. Bosman. Learning and anticipation in online dynamic optimization. Evolutionary Computation in Dynamic and Uncertain Environments, Springer Verlag, Berlin, pp. 9 5, 007. [] J. Branke. Memory enhanced evolutionary algorithms for changing optimization problems. Proc. of the 999 Congress on Evol. Comput., pp , 999. [] J. Branke, T. Kaußler, C. Schmidt, and H. Schmeck. A multi-population approach to dynamic optimization problems. Proc. of the Adaptive Computing in Design and Manufacturing, pp , 000. [5] J. Branke. Evolutionary Optimization in Dynamic Environments, Kluwer Academic Publishers, 00. [6] R. Fitch, B. Hengst, D. Suc, G. Calbert, and J. Scholz. Structural abstraction experiments in reinforcement learning. AI 005: Advances in Artificial Intelligence, Springer Verlag, Berlin, pp. 6 75, 005. [7] Y. Jin and J. Branke. Evolutionary optimization in uncertain environments a survey. IEEE Trans. on Evol. Comput., 9(): 0 7, 005. [8] E. H. J. Lewis and G. Ritchie. A comparison of dominance mechanisms and simple mutation on nonstationary problems. Parallel Problem Solving from Nature PPSN V, Springer Verlag, Berlin, pp. 9 8, 998. [9] R. W. Morrison and K. A. De Jong. Triggered Hypermutation Revisited. Proc. of the 000 Congress on Evolutionary Computation, pp. 05 0, 000. [0] H. Richter. A study of dynamic severity in chaotic fitness landscapes. Proc. of the 005 IEEE Congress on Evolutionary Computation, pp. 8 8, 005. [] R. Tinós and S. Yang. A self-organizing random immigrants genetic algorithm for dynamic optimization problems. Genetic Programming and Evolvable Machines, 8(): 55 86, 007. [] S. Yang. Associative memory scheme for genetic algorithms in dynamic environments. Applications of Evolutionary Computing, Springer Verlag, Berlin, pp ,

Learning the Dominance in Diploid Genetic Algorithms for Changing Optimization Problems Shengxiang Yang Abstract Using diploid representation with dominance scheme is one of the approaches developed for

HYBRID GENETIC ALGORITHMS FOR SCHEDULING ADVERTISEMENTS ON A WEB PAGE Subodha Kumar University of Washington subodha@u.washington.edu Varghese S. Jacob University of Texas at Dallas vjacob@utdallas.edu

PLAANN as a Classification Tool for Customer Intelligence in Banking EUNITE World Competition in domain of Intelligent Technologies The Research Report Ireneusz Czarnowski and Piotr Jedrzejowicz Department

Chapter 5 Clustering & Visualization Clustering in high-dimensional databases is an important problem and there are a number of different clustering paradigms which are applicable to high-dimensional data.

American Journal of Industrial and Business Management, 2016, 6, 774-789 Published Online June 2016 in SciRes. http://www.scirp.org/journal/ajibm http://dx.doi.org/10.4236/ajibm.2016.66071 A Study of Crossover

A New Multi-objective Evolutionary Optimisation Algorithm: The Two-Archive Algorithm Kata Praditwong 1 and Xin Yao 2 The Centre of Excellence for Research in Computational Intelligence and Applications(CERCIA),

Senior Secondary Australian Curriculum Mathematical Methods Glossary Unit 1 Functions and graphs Asymptote A line is an asymptote to a curve if the distance between the line and the curve approaches zero

NEW YORK STATE TEACHER CERTIFICATION EXAMINATIONS TEST DESIGN AND FRAMEWORK September 2014 Authorized for Distribution by the New York State Education Department This test design and framework document

6.2.8 Neural networks for data mining Walter Kosters 1 In many application areas neural networks are known to be valuable tools. This also holds for data mining. In this chapter we discuss the use of neural

CURVE FITTING LEAST SQUARES APPROXIMATION Data analysis and curve fitting: Imagine that we are studying a physical system involving two quantities: x and y Also suppose that we expect a linear relationship

On the representability of the bi-uniform matroid Simeon Ball, Carles Padró, Zsuzsa Weiner and Chaoping Xing August 3, 2012 Abstract Every bi-uniform matroid is representable over all sufficiently large

KY San Jose State University Engineering 10 1 Select Insert from the main menu Plotting in Excel Select All Chart Types San Jose State University Engineering 10 2 Definition: A chart that consists of multiple

Compact Representations and Approximations for Compuation in Games Kevin Swersky April 23, 2008 Abstract Compact representations have recently been developed as a way of both encoding the strategic interactions

3. Reaction Diffusion Equations Consider the following ODE model for population growth u t a u t u t, u 0 u 0 where u t denotes the population size at time t, and a u plays the role of the population dependent

123 CHAPTER 5 CORONARY ANGIOGRAM VIDEO COMPRESSION USING WAVELET BASED CONTOURLET TRANSFORM AND REGION OF INTEREST TECHNIQUE 5.1 INTRODUCTION Coronary angiograms play an important role in the diagnosis

2 Exercises and Solutions Most of the exercises below have solutions but you should try first to solve them. Each subsection with solutions is after the corresponding subsection with exercises. 2.1 Sorting

University of Babylon Undergraduate: third class College of information technology Department of software Subj.: Application of AI lecture notes/2011-2012 ***************************************************************************

Competitive Analysis of On line Randomized Call Control in Cellular Networks Ioannis Caragiannis Christos Kaklamanis Evi Papaioannou Abstract In this paper we address an important communication issue arising

CS/COE 1501 http://cs.pitt.edu/~bill/1501/ Lecture 01 Course Introduction Meta-notes These notes are intended for use by students in CS1501 at the University of Pittsburgh. They are provided free of charge

Artificial Neural Networks and Support Vector Machines CS 486/686: Introduction to Artificial Intelligence 1 Outline What is a Neural Network? - Perceptron learners - Multi-layer networks What is a Support

International Journal of Scientific and Research Publications, Volume 3, Issue 3, March 2013 1 An Efficient Majority Logic Fault Detection to reduce the Accessing time for Memory Applications R.Meenaakshi

Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear