CYRUS 2D Simulation Team Description Paper 2017

Nader Zare, Ali Najimi, Mahtab Sarvmaili, Aryan Akbarpour, Mohsen NaghipourFar, Borna Barahimi, Amin Nikanjam

Dept. of Computer Engineering, Khajeh Nasir University of Technology; Dept. of Computer Engineering, Sharif University of Technology; Bellerbys College Brighton; Atomic Energy High-school

Abstract This paper includes some explanations about algorithms implemented by CYRUS team members. The main objective is to express a brief explanation about intelligent decision making of the agents. For this purpose, first we applied deep neural network for opponents position detection, and also de-noising and agents restricted view removal, and second, another deep neural network were learned to find the opponent team formation. These two neural networks enable our agents to understand the opponents team formation and also improve their performance by de-noising and removing view limitations. The base code used by CYRUS is agent 3.1.0[1].

Introduction

"CYRUS" robotic team members created this team six years ago, with the goal of student scientific improvement in Artificial Intelligence and multi-agent field. At first, members of this team were bachelors of Information Technology Dept. at Shiraz University of Technology and now team members are formed of students from Khajeh Nasir University of Technology, Sharif University of Technology, and Atomic Energy High-school.

This team has been qualified to participate in World Competition since 2013 and ranked 8th, 5th, 9th, 12th (CYRUS) and 8th (FURY) sequentially, Also in 2014 achieved First-place of Iran Open competition, first-place in Kordestan 2013 and first-place in Fazasazan 2012. This team has also participated in Iran Open 2013, Iran Open 2012, SharifCup 2012, Iran Open 2011, Sama RoboCup 2011, etc. Presently CYRUS team members are trying to improve their efficiency by using Machine Learning, and multi-agent algorithms.

This paper briefly represents CYRUS team structure and algorithms, e.g. opponent behavior detection, and opponent team formation detection using deep learning.

Opponent Behavior Detection

Some of the important aspects of 2D Soccer Simulation are noise, agents restricted view, and opponents behavior detection in the context of uncertainty. CYRUS team uses deep learning from the Robo Cup logs from 2010 to 2016 in order to detect these behaviors.

In this section our purpose is to determine opponents next position by current positions of our team and opponent. At the moment, learning process is only available offline but in the future, online learning process will be added to coach.

Data-Set

Deep neural networks need large amounts of data for training, to meet this requirement over 15 million data have been gathered from Robo Cup logs since 2010 to train deep neural network.

Each record consists of 144 features which 94 features of them have been considered as inputs to the neural network. Useful information of both teams and ball position of every single cycle (assume as cycle i) are given to the neural network as inputs, and information of the next cycle (assume as cycle i+1 ) are stated as labels.

List of inputs

List of labels

Using Stacked (De-noising) Auto-Encoders to Find The Best Initial Weights

The dominant method for training deep neural networks is gradient descent. Training deep neural network from random initialized parameters, does not work very well, and primary layers don't learn properly. Few algorithms work well for this purpose. Learning layer by layer using Auto-Encoder has been implemented, which weights each layer learns separately to have better initialization for each individual layer so that it may learn useful feature extractors. After building the stack of encoders, the supervised learning can be used to learn for training the deep neural network layers. Auto-encoder sets output label of each layer equal to input of it in order to make distinction between different data and to be noise resistant.

Each layer includes respectively 200, 150, 100, 80, and 48 neurons. Methods of achieving initial weights, will be defined.

Finding The Best Structure to Learn Neural Network

To achieve the efficient initial weights, Multi-Layer Perceptron (MLP) and Rough neural networks have been implemented.

Rough neural networks cleave weights into lower bound and upper bound, then from each part of weights output will be averaged.

$$MLP:$$

$$Net = W_X + b \longrightarrow O = f(Net)$$

Rough :

$$Net_{U} = W_{U} * X + b_{U}$$

$$\downarrow \qquad \qquad \downarrow$$

$$O_{U} = max(f(Net_{U}), f(Net_{L}))$$

$$Net_{L} = W_{L} * X + b_{L}$$

$$\downarrow \qquad \qquad \downarrow$$

$$O_{L} = min(f(Net_{U}), f(Net_{L}))$$

$$W_2 = transpose(W_1) \tag{1}$$

Learn W^1 and gain W^2 from W^1

$$W_1 \parallel W_2 \tag{2}$$

Learn W^1 and W^2 simultaneously

$$W_1 = transpose(W_2) \tag{3}$$

Learn W^2 and gain W^1 from W^2

Equations [1] and [2] and [3] are methods to achieve efficient initial weights according to the layer type (MLP or Rough) diagrams are shown in Fig. 3.

Learn Deep Neural Network Using Mined Initial Weights

To detect opponents next cycle behavior, a 7-layers deep neural network with 200, 150, 100, 80, 48, 48, 48 neurons sequentially were used in this algorithm that initial weights can be assigned randomly or by using mined weights from above methods, are shown separately in Fig. 4, and CYRUS team uses the best result.

Using Deep Learning to Detect Opponents Next Cycle Behavior

Currently, CYRUS team can detect opponents next cycle behavior using defined deep neural network, and in the future, it will detect next cycles by adding self positions. For instance, Cycle i information will be given to the neural network and cycle i+1 will be our result, then cycle i+1 information will be given to the neural network and cycle i+2 will be our result. This operation will be continued until the desired information is achieved. Table 3 shows the result of behavior detection.

Average scores with and without neural network

Formation Detection

Formation is the most distinctive feature of every 2D Soccer Simulation team and it consists of different lineups e.g. defense, midfield, and offense. CYRUS team will form its best formation against different teams and formations using opponents formation detection and Artificial Intelligence methods, that will be defined, before Robo Cup.

In order to determine opponents formation, first a great data-set of 80 different formations were created using fedit[2]. Each and every data from data-set includes: ball position, positions of all the agents, angle and direction between ball and every agent as inputs of the neural network and number of agents in each line as labels of inputs. The learning process uses a 6-layer neural network which contains 25, 80, 50, 30, 15, and 5 neurons sequentially. The initial weights are calculated from auto-encoder method. Fig. 5 shows total error and accuracy according to the epochs in the learning process.

A teams formation may be different in viewpoint of person to person, it is not possible to compare with reality, Therefor coach gathers 500 cycles of information then gives it to neural network in two different ways:

• (1) each cycles information will be given to neural network and final result will be averaged.
• (2) all information will be averaged then will be given to neural network.

Accuracy of each method for agent2d formations

Future

CYRUS team members plan to enhance offense and defense system using deep neural networks, and deep reinforcement learning. In order to achieve this goal, CYRUS team will reprogram its defense real-time decision making using actorarctic deep neural network and multi-agent reinforcement learning, goalie decision making in penalty kicks using actor-arcitic multi-agent deep learning and deep reinforcement learning, online opponents high level behavior detection using PGM and deep learning with coach.

References

1. "agent2D-3.1.0 RoboCup tools - OSDN." [Online]. Available: http://en.osdn.jp/projects/rctools/downloads/51943/agent2d-3.1.0.tar.gz . [Accessed: 22-Jan-2016].
2. "fedit2-0.0.0 RoboCup tools - OSDN." [Online]. Available: https://osdn.net/projects/rctools/downloads/48791/fedit2-0.0.0.tar.gz/
3. Teshneh Lab, Mohammad, and Pouria Jafari. Neural Networks and Advanced Neuro-Controlers, A Rough Neural Network Approach. Khajeh Nasir University of Technology, 2015.
4. Haykin S., "Neural Networks: A Comprehensive Foundation (2 ed.)", Prentice Hall, 1998.
5. H. Akiyama, T. Nakashima, and S. Mifune, "HELIOS2015 Team Description Paper," pp. 16, 2015.
6. Vincent, Pascal, et al. "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion." Journal of Machine Learning Research 11.Dec (2010): 3371-3408.
7. Slotine, Jean-Jacques E., and Weiping Li. Applied nonlinear control. Vol. 60. Englewood Cliffs, NJ: Prentice-Hall, 1991.
8. Akiyama, H., Noda, I.: Multi-agent positioning mechanism in the dynamic environment . In: Visser, U., Ribeiro,F., Ohashi, T., Dellaert, F., eds.: RoboCup 2007: Robot Soccer World Cup XI, Lecture Notes in Artificial Intelligence. Volume 5001., Springer (2008) 377-C384.
9. N.Zare, A.Keshavarzi, E.Beheshtian, H.Mowla, H.Jafari, "CYRUS 2D Simulation Team Description Paper 2016", The 20th annual RoboCup International Symposium, Germany, Leipzig, 2016.
10. N.Zare, M.Karimi, A.Keshavarzi, E.Asali, H.Alipour, A.Aminian, E.Beheshtian, H.Mowla, H.Jafari, M.J.Khademian, "CYRUS 2D Simulation Team Description Paper 2015", The 19th annual RoboCup International Symposium, China, Hefei, 2015.
11. R.Khayami, N.Zare, M.karimi, P.Mahor, F.Tekara, E.Asali, A.Keshavarzi, A.Afshar, M.Asadi, M.Najafi, "CYRUS 2D Simulation Team Description Paper 2014", The 18th annual RoboCup International Symposium, Brazil, Joao Pessoa, 2014.
12. M. Dezfoulian, N. Kaviani, A. Nikanjam, M. Rafaee, Training a Simulated Soccer Agent how to Shoot Using Artificial Neural Network, 13th Multi-disciplinary Iranian Researchers Conference in Europe (IRCE), Leeds, UK, 2005.
13. M. Prokopenko, P. Wang, and O. Obst, "Gliders2015: Opponent avoidance with bio-inspired flocking behaviour," pp. 15, 2015.
14. H. Zhang, M. Jiang, H. Dai, A. Bai, and X. Chen, "WrightEagle 2D Soccer Simulation Team Description 2015," Wrighteagle.Org, pp. 38, 2015.
15. S. Marian, D. Luca, B. Sarac, and O. Cotarlea, "OXSY 2015 Team Description," 2015.