View Original PDF

SBCS Team Description Robocup 2004 Simulation League

Eslam Nazemi, A. Jorati, V. Kazemi, Z. Parkam

Computer Science Department - Shahid Beheshti University Evin, Tehran, IRAN

http://www.sbcee.net

Abstract This paper is written as much to fulfill qualification acquirements for participation in the 2004 International Robot Soccer World Cup, as to represent the result of the extensive research newly conducted in the Computer Science Department mainly concerning relevant areas of AI, such as Reinforcement Learning during the last year.

1 Introduction

The SBCS Soccer Simulation team is built by three junior undergrad CS students. As a first time participant in the soccer simulation league and due to its challenge, although some of team members brought valuable expriences from other teams such as SBCe2003, we based our team on the high performance team UvA Trilearn 2003 so that we could focus on particular details rather than to build the team from scratch. Besides we borrowed some ideas, which we have improved to some extent from the two great teams, namely CMUnited'99 and FC Portugal 2000. We actually did great enhancements to them.

2 Team Development

Our newly founded team consists of:

A) Team Leader:

Eslam Nazemi

  • Faculty member

B) Team Members:

Amin Jorati

Vahid Kazemi

Zabiholla Parkam

  • Undergraduate students
  • Members of a research group in Machine Learning

3 Agent Architecture

Our main objective in the SBCS 2004 team is to build a firm research platform for multiagent systems research. Currently, a multi-threaded approach is adopted to achieve real time performance and a modular approach is adopted in the overall agent implementation.

3.1 Low- and High-Level Behaviors

The Agents feature a set of low-level and high-level behaviors. At the lowest level any decisions made by the agent must be reduced to the standard actions the server provides, i.e. Kick, Turn and Dash. In order to provide some options for high-level behaviors, we made enhancements to the server standards, such as:

  • Move: In case an agent wants to move, it must mix turns and dashes and maybe avoid collisions with other objects to reach the desired location. And the high-level tactical behaviors are built on top of low-level primitive behaviors, and are currently implemented as a hybrid of Q-learning and rule based decision-making.
  • Dribble: Dribbling involves intermixing kicks, dashes, and possibly turns to move the ball across the pitch, while keeping the ball under the agent's control.

4 Reinforcement Learning-based features

We found it helpful to have fixed plans for situations which happen frequently. SBCS 2004 team has implemented some of the fixed plans in RoboCup2003. The framework for our fixed plans consists of three major components, namely Triggers, Actions and Abort conditions. The Triggers is used as signals to allow or forbid the actions or plans that are predefined. The Actions is a combination of low-level and high level behaviors that are executed sequentially. The Abort condition is a safe guard to ensure the conditions of the environment are suitable for the fixed plan being executed. We have adopted reinforcement learning in our behavior-based decision making process since it provides a way to programming agents by reward and punishment without needing to specify how the task is to be achieved . Each time the agent receives inputs, it then chooses an action to send. The action changes the state of the environment and also provides the agent with either a reward if it does well or a punishment if it badly. The agent should choose actions that maximize the long-term sum of rewards. It should be noticed that the agents in our implementation not only have different roles and responsibilities, but also have different sub-goals in the team. Hence, every individual agent tries to reach its own goal state, and cooperation emerges when the goals of all agents are linked together. The ultimate goal, scoring against the opposition, becomes a joint effort that is distributed among team members.

In building our team, we reused some low-level solutions from the publicly available UvA-2003 team and have made several significant enhancements to it. While creating SBCS04, we pursued two some major goals: most importantly processing with great precision the information received from the environment and the coordinate information and also to improve the agent/player behavior model.

The first direction of our research has ushered in the better accuracy of agent selforientation and moving object parameter estimation algorithms. This resulted in a statistically significant score difference gain while the enhanced UvA team was playing against the real one. .

6 Future Work

Now that we have come so far, and we've seen our reinforcement learning-based algorithms in action, we will base our future work on advanced researches in the area of reinforcement learning ,which has proved to be successful in high level decisions of multi-agent team like co-planning and positioning. We will in the meanwhile try to implement different policies and get our theaoretical results down into action to adapt them into the robocup aims, such as better defining state spaces,and much more complex issues.

References

  1. Balch T.R., Integrating RL and Behavior-based Control for Soccer, Proc. IJCAI Workshop on RoboCup, 1997.
  2. Andrew G. Barto,Richard S. Sutton, Reinforecement Learning :An Introduction. MIT Press,1998
  3. R. de Boer & J. Kok, The Incremental Development of a Synthetic Multi-Agent System: The UvA Trilearn 2001 Robotic Soccer Simulation Team. M.S. Thesis. Faculty of Science, University of Amsterdam, 2002.
  4. Hunter M., Kostiadis K., Hu H., "A Behaviour-based Approach to Position Selection for Simulated Soccer Agents", 1st European Workshop on RoboCup, Amsterdam, 28 May - 2 June 2000
  5. Kostiadis K. and Hu H., Reinforcement Learning and Co-operation in a Simulated Multi-agent System, Proc. of IEEE/RJS IROS'99, Korea, Oct. 1999.
  6. Stone, Peter , Layered Learning In Multiagent Systems : A Winning Approach To Robotic Soccer. MIT Press, 2000.
  7. Stone Peter and Veloso M., Team-Partitioned, and Opaque-Transition Reinforcement Learning, Proc. 15th International Conference on Machine Learning, 1998.
  8. RoboCup 2001: Robot Soccer World Cup V. Andreas Birk … (ed.) Berlin et al. Springer, 2002.