LegenDary 2013 Soccer 2D Simulation Team Description Paper

Pourya Saljoughi, Narges Majidi, Reza Ma'ani jou

Department of Computer Engineering, Guilan Univercity, Rasht, Iran

Abstract In LegenDary project, we have been continuing to develop our team based on Agent2D to achieve our goals and purposes. This year, we have optimized our shoot system by using Reinforcement Learning. We have improved our defense system by adding a new Marking system. In this paper, we are going to briefly describe our latest achievement to these tasks and strategies. Keywords: 2D Soccer Simulation, Reinforcement Learning, Dynamic Shoot Algorithm, Mark

Introduction

LegenDary was qualified for the first time at RoboCup 2012, but because of financial problem of our previous affiliation, we couldn't participate at the competition, and we sent our withdrawal email to the Technical Committee before deadline and it was accepted. This year, we have solved our last year's problems and we hope to participate in RoboCup 2013 competitions.

LegenDary project was started on UVA base source code [1]. After competing in some competitions we have changed our base to Agent2D [2] [3]. Before this year, almost all skills and methods of LegenDary was using static algorithms. This year we have focused on adding learning algorithms to our agents in order to develop a more realistic soccer team. To achieve this goal, recently we have added a simple reinforcement learning algorithm [4] to LegenDary's shoot skill besides improving other skills. Although we have still many instabilities in our skills, we hope that until this year's RoboCup we can provide a stable and competitive team to take part at the competitions.

Our paper is organized as follows. In Section 2 our new marking system named "GreedyMark" is introduced. Section 3 briefly describes our improved shoot algorithm with reinforcement learning with experimental results after implementing this algorithm. At last our future works and conclusion of this paper can be found in Section 4.

Greedy Mark

In defense situations, generally defenders will decide to block the opponent's ball owner or go to their strategic position or mark the offense players. In LegenDary we have implemented a new method this year for defensive situations in order to mark the opponent players in a way that they can't receive a pass until the ball is sent out of danger area for our defense. The idea came from team BrainStormers' defense system which was based on a strict marking system and when they started to mark, their marking system would not stop until a safe position for their team have happened [6]. In the next paragraph the algorithm of our newly implemented marking system is discussed.

Decision process will begin when the ball comes to one of our dangerous areas and it is affected by some parameters like our teammates' stamina, their positions, and their post in the field. Then a priority list of them will be created and first players in the list are the first players that should start marking opponents. Each one of our players must mark one of the opponents players like an aim for them and their aims are different from each other. This algorithm is designed in order to keep the ball away from dangerous areas that scoring a goal against our team is possible. And simultaneously blocker teammate (or teammates) will have more time doing its job to force the ball owner in order to make him send a backward pass and send the ball out of our dangerous areas or to get the ball from that agent.

We have called this algorithm greedy because when one of our teammates aims to mark one of the opponents it won't let go of that opponent even for a single cycle and physical marking will strictly execute considering defender's stamina until one of the strict boundary conditions happen. In below, We show one example of Greedy Mark in figure 1.

Dynamic Shoot

Static algorithms are good for many situations and may lead to a good result against average teams. These algorithms may also perform well against powerful teams but they have limitations in quality. If in one specific situation against one opponent team the result of execution of a static algorithm was not good, that mistake will happen every time code is executed in the same situation. This is a problem of Model-based agents[5]. We can improve our agent's performance by Reinforcement Learning algorithm in order to have a learning-based agent[5]. Our agent will prevent repeating the same mistake in the same situation. LegenDary's skills have been using static algorithms before this year but now we are improving our algorithms by using Reinforcement Learning. Our agents try to improve their performance by using their previous observation so we don't see same mistakes every time. In our first step of adding learning algorithm to our team, we have improved our shoot algorithm by Reinforcement Learning and after shoot skill, we will expand it to our other skills such as pass and formation.

As we mentioned before, using Reinforcement Learning is our goal for improving our team. For achieving this goal we added Reinforcement Learning to our team's shoot algorithm. In this algorithm, at first a suitable point for shoot is selected by using "dynamicPredictShootPoint" function, after that the agent performs shoot action.

Our previous predict shoot point function has static weights for choosing a suitable point for shooting toward the goal. These weights may not work in front of all goalkeepers and teams with different levels of power. Although it works in front of average teams but it's not good enough for some situations and shoot might be missed in some important situations. On the other hand the type of the goalkeeper is also important and the operation of this function varies from type to type in some parameters.

In our new algorithm, at the beginning of the match our agents use static weighting for prediction as a base, but after performing each shoot, it will observe the result until the ball enters the goal or it is catched by a player or the goalkeeper. In each situation the result will be observed and saved for that player. Next time, in which shoot is going to be executed, weighting is changed by considering previous observation of the agent and after that, the agent searches the points and chooses the best point with new updated weights and this process will continue each time a shoot is performed.

After optimizing LegenDary's shoot with Reinforcement Learning and testing it against goalkeepers from different teams in one-on-one situation when the game is in play_on mode, a good result was achieved as shown in the following table. Percentage of the goals in one-on-one situation with dynamic shoot algorithm is available in figure 2.

Experimental results of RL improved shoot algorithm.

Future Works

This paper briefly described some parts of recent works in LegenDary project and our recent achievements since our last TDP [3]. This year we have focused on adding simple AI algorithms to our team besides improving our existing skills. And in next step we will extend using learning algorithms such as using artificial neural networks for our formation and pass system and simultaneously improving existing skills.

References

1. UvA Trilearn 2003. http://www.science.uva.nl/˜jellekok/robocup/2003/
2. Agent2D source: http://sourceforge.jp/projects/rctools/
3. Saljoughi, P., Ma'anijou, M., Fouladi, H., Majidi, N., Yaghoubi, S., Fallah, H., Zahedi, S.: LegenDary 2012 Team Description Paper. In: RoboCup 2012 Symposium and Competitions, Mexico (2012)
4. Pack Kaelbling, L., L.Littman, M., W.Mooore, A.: Reinforcement Learning: A Survey. In: Journal of Artificial Intelligence Research 4 (1996) 237-285.
5. J. Russel, S., Norvig, P.: Artificial Intelligence A Modern Approach Second Edition. In: Chapter 2, page: 48-54. (20030)
6. Riedmiller, M., Gabel, T.: Brainstormers 2008 Team Description Paper. In: RoboCup 2008 Symposium and Competitions, China (2008)