AUA2D Soccer Simulation Team Description Paper for RoboCup 2011

Lei Tao, Runmei Zhang

School of Electronics and Information of Engineering, Anhui University of Architecture, China

Abstract This paper briefly describes our new techniques both in high level and low level, which we apply to AUA2D 2011. Our Q-learn algorithm is based on the improvement of the LCS algorithm and the team's combination of offensive and defensive systems, mainly for multi-agent to cut-off and pass the ball , to improve the learning ability of the agent. We also strengthen the defensive strategy to avoid the opponent Special Skills ball, make an improvement of one-on-one against model to improve the the initiative of agents. Many improvements and innovations apply to AUA2010, at the same time we gradually use the agent2d underlying code in current research.

1 Introduction

The AUA RoboCup Team was established in 2003, starting with the 2D soccer simulation team only. In the following years, the 3D soccer simulation team, MSRS team and Rescue simulation team have joined the AUA. Since 2003 We participated actively in RoboCup China Open and obtained good result. AUA 2D took the 7th place in RoboCup China Open 2008.

In 2010, we have developed a new team structure based on the BP algorithm. And we have participated the WorldCup in Singapore, we study the learning ability of agents which has been a significant improvement to the team strength. This paper briefly describes our new techniques both in high level and low level, which apply to AUA2D 2011 .

2 Learning Ability

LCS (Learning Classifier System) is applied to our passing strategy, this method is much less than the required storage space in general Q-learning.

In this system, there are two parallel reinforcement learning algorithms and the reinforcement learning algorithm is for discovering rules or the classification used to determine the weight.

The system can be divided into three subsystems, execution subsystem, weight distribution subsystem and rule discovery subsystem, which intend to complete the team in passing.

2.1 An Execution Subsystem

The system is mainly to help players to pass the ball more efficiently, there are six components:

• 1. Message List(ML) The main effect is to accept the current environmental information and their own internal information from the server, the agent module can be coded to corresponding action( shooting, passing, ball, catch the ball, chase the ball, etc. ).
• 2. Action Module This module is for agents to get a good agent moves the current coding pass code matches the action, and output good pass encoding, through the selection of weights according to their own learning in the race.
• 3. Classifier Set This module is stored by the implementation of goals per agent action rule to improve the learning ability of the team.
• 4. Rules of the Constructor Function Rules of the constructor function is constructed which based on motion information of the robot, we used a new rule in rule set ,when the match rules are not available, the module will randomly choose the implementation of a pass code and send to the actuator movement, then the rules to generate new action constructor Rules.
• 5. Match Classifier Set Match Classifier Set is a collection of fixed size whose elements are the list of messages with the current rules in it, matching certain rules.
• 6. Conflict Resolution Module Conflict resolution module action under the rules of each select an action code weights, in some cases, matching rules set is empty, then the implementation of the module will randomly choose a pass code and send it to the actuator movement.

2.2 Weight distribution subsystem

The subsystem is actually closely linked with the implementation of subsystems, each subsystem in the implementation cycle, if the LCS by the reward or punishment, the value of the incentive effect of distribution subsystem caused by the action of the match in the rule set Rules, by winning their weight or reduce the punishment such as the agent of our goal, the reward for passing the rules, if our agent conceded, pass rules will be punished.

2.3 Rule Discovery Sub-system

Rule discovery sub-system function is the optimum from the existing rules in the rules to derive new rules, all rules focus on the individual genetic algorithm, the rules in the rule set to determine the weight, the genetic algorithm. These rules are less removed from the rules, then the rules for optimum operation of crossover and mutation to obtain a new set of rules back into the rules, start the next round of enhancements focused on learning so as to continuously optimize the pass rules improving the accuracy of passes.

Passing in the implementation of the rules introduced in the module constructor, passing the constructor function of the rules is based on motion information of agents commonly used to construct a new pass rules, on the one hand make full use of agent movement information for each step, we make use of multi-agent systems feature parallel to accelerate the learning process agent.

3 Introduction of Team Defense System

• 1. Defense capabilities in the team:our team's defenses can be divided into five parts, the first field defense, midfield defense, backfield defense, defense dangerous areas, and defense against many teams special skills attack, and in defensive part we use them to increase part of the different algorithms.
• 2. The defensive zone is used to enhance the defensive ability of agents to prevent the making of the jitter, we are in the code in the agents manmark, not only to the opponent's weight in order of importance, the only opposition player to consider is the importance of field position, we could consider agents from their own agents and opponents of the importance of the opponents, you can also consider the opposition player to intercept the ball that the number of cycles of the importance of the opponents.

4 One-to-One Against Model

While watching video and analysing the underlying code in the team competition with a number of strong teams, I found a problem in the opponent restricted areas, the agent lose the ball easily against each other.

To be able to drive the ball in the smooth process of the other agents to avoid, we added a one-on-one against model code, in front of opponent gate, set more passing paths ,and Semi-MDP algorithm apply to one-to-one against model, mainly to help training a good player, mainly through local modeling of opponents.

5 Players Dribble

Our offensive strategy is the most used players dribbling through the visual ability of team learning, and in terms of dribbling line heuristic search through a certain dash the ball to find the best route.

6 Conclusion and Future Work

This year, we need to do these things: first we need to increase is the collaboration between agents and enhance online learning strategies of competitors; second, we need to build the opponents model, which predict behaviors of the opponents; third, training for agents as we need, and searching for a more attacking line route.

References

[1] Runmei ZHANG, Hongliang YAO. Separators Introduced BK Inference Algorithm and its Application in RoboCup. Computer Science.Vol.36(6).June,2009. [2] Master thesis, Chinese Academy of Sciences Institute of Automation. About Multi-robot based on improved learning algorithm for LCS, 2010 [3] Bachelor thesis, Central South University, About Defense system, part of the algorithm,2008. [4] Xiaoping Chen, et al, Challenges in Research on Autonomous Robots, Communications of CCF,Vol.3,No.12,Dec,2007. [5] Gang WANG, Mubin CHEN, Fuhong LIANG, Shumei ZHENG. A Study of the Passing Strategy on the RoboCup Simulation Game. Computer Engineering and Science.Vol.29(10).Oct,2007. [6] Satje A. Reintbmement Learning of Player Agents in RoboCupSoccer Simulation[A]. Proc of the 4th IntI Conf on Hybrid Intelligent Sys-tem[C].2004.