a4 ty'2003 Team Description
Alexey Vasilyev, Andrei Kapishnikov, Anatoly Sukov
Riga Technical University, 1 Kalku street, LV-1658 Riga, Latvia
Abstract a4 ty'2003 soccer simulation team's main research focus is online adaptation and machine learning. We created an adaptive system that uses a counterpropagation artificial neural network in decision-making. The network is trained by a specially designed learning algorithm.
Introduction
A huge number of states and possible policies makes the RoboCup simulation environment [1] extremely popular for different heuristic learning systems. Although the RoboCup environment allows using real-time (online) learning, nevertheless the suppressing majority of RoboCup agents use offline training as a basis for decisionmaking. This means that the agents are trained to react to every situation determined and do not change their policy of behavior in the process of a game. However, it is highly possible that during the game against a given opponent, it will found out that the behavior of the agent is not correct in some situations. In this case, the online adjustment of this behavior is needed. Moreover, (1) the number of iterations necessary for retraining should be minimal; and (2) the agent should manage in time, which is given him for decision-making on each game-cycle (usually 100 ms).
Online adaptation
In a4 ty team we proposed using of counterpropagation artificial neural network [2] with specially designed algorithm [3], as one of the possible ways of implementation of online adaptation. This approach allows creating an agent that can quickly adapt to the surrounding environment on the basis of the trial-and-error method which, in turn, enables one to use it for a real-time adaptation in the RoboCup simulation environment.
Counterpropagation neural network consists of two layers of neurons: a Kohonen self-organizing feature map and a Grossberg star. The layer of the Kohonen neurons allows creating classes of input patterns and then relates a newly presented pattern to one of the classes. The layer of the Grossberg neurons is used for forming the required reaction to each pattern. A new learning algorithm changes the weights of the Grossberg neurons in online mode using a reinforcement signal.
We applied this approach for pass selection. A trained, with offline learning, agent needs to decide whether it should give a pass to a partner or not. If it makes an incorrect pass (the ball is intercepted by an opponent) in online mode in the given situation, the network updates weights so as next time the probability of choosing this action for the same (or similar) situation would be lower.
However, this approach has not been implemented in the final version of a4 ty team since the overall performance (approximation capabilities) of counterpropagation network is worse than, for example, multilayer perceptron has. Thus, counterpropagation neural network has been replaced by multilayer perceptron (MLP).
Offline adaptation
The main advantage of the MLP is that it can successfully be applied for approximation of conditional probability function [4]. Such an approach makes it possible to examine the training sample with contradictory vectors and to interpret the outputs of the network in classification tasks as the evaluation of the probability of belonging to the specific class. This feature is extremely useful in uncertain situations that RoboCup environment has.
Two offline trained neuron networks has been employed for calculating a probability for scoring the ball in the given situation and a probability for successful passing.
Conclusion and future work
In this paper we have presented a brief description of the a4 ty RoboCup simulation league team with the main research contribution made during the team's development. The code of the a4 ty'2003 team is based on UvA Trilearn basic agent [5].
The future work will be concerned with the development of new machine learning algorithms for decision-making and an intelligent online trainer. The current version of the trainer uses only self-organizing feature map for opponent's formation analysis.
References
- Mao Chen, Klaus Dorer, Ehsan Foroughi, Fredrik Heintz, ZhanXiang Huang, Spiros Kapetanakis, Kostas Kostiadis, Johan Kummeneje, Jan Murray, Itsuki Noda, Oliver Obst, Pat Riley, Timo Steffens, Yi Wang, and Xiang Yin. RoboCup Soccer Server User Manual: for Soccer Server version 7.07 and later, 2002. At http://sourceforge.net/projects/sserver.
- Hecht-Nielsen R. Counterpropagation networks. In Proceedings of the IEEE First International Conference on Neural Networks, eds. M. Caudill and C. Butler, vol. 2, pp. 19- 32. San Diego, CA: SOS Printing, 1987.
- Alexey Vasilyev, Andrei Kapishnikov, and Anatoly Sukov, Quick Online Adaptation with Reinforcement of Simulated Soccer Agents. In press, 2003.
- Alexey Vasilyev, Andrei Kapishnikov, Approximation of Conditional Probability Function Using Artificial Neural Networks. In Proceeding of International Conference on Modelling and Simulation of Business Systems - MOSIBUS'2003, Vilnius, Lithuania, May 13-14, pp.79-81, 2003, ISBN 9955-09-420-6.
- Remco de Boer & Jelle Kok, The Incremental Development of a Synthetic Multi-Agent System: The UvA Trilearn 2001 Robotic Soccer Simulation Team. Master's thesis, University of Amsterdam, 2002.