PNU_S3D Team
Mohammad Massah Bavani, Hamid reza sharifi, M. baghery
Iran Esfahan Payam Noor University
Abstract This paper describes the main ideas and strategies of pnu_s3d team . In this page we explain : our team formation , agent Localization methods , reinforcement learning of team strategies , and the base layers of our team .
Introduction
The Robocup Simulation League 3D Soccer Competition has been held for the first Time in Robocup2004 . which is an important milestone on the path to the ultimate goal of the Robocup .The Server of Soccer3D (rcssserver3d ) is a Multi-Agent simulations system for Physical agent in three dimensional environment and resembles to the real world much better than 2D.This Server is based on SPADES( system for parallel Agent Discrete Event Simulation ) which is a middleware system and has unique properties; it is an event-based system that can be distributed among several machines . Rcssserver3D consider real phy sical rules like friction and gravity.So calculation of some facts like prediction of position and velocity of ball and agent or interception are much more complex .[5]
Simulated envirroments are a commonly used method for researching artificial intelligence Methods in physical multi-agent system.Simulations are especially useful for two different Types of problems : (1) to experiment with different sensors,actuators or morphologies of Agents and (2) to study team behavior with a set of given agents. Additionally the connection Between both types of problems is an interesting research problem[1].
According to our experience in developing a team of intelligent agents , we have come to A customized methodology and approach to construct an extensible , reusable and robust Framework.
Localization
The localization of agents is based only on the vision perception about the flags that agents receives from the server in each cycle . The flags are not moving so the server noise is very Little for them and they are the best guidence for Localization . The agent has 360 degree view of the field and no direction,so location can be computed Using relative position of only one flag.However, because the agent percepts each falgs with Different amount of noise in each cycle(server generates noise for each flag independently)We used all 8 flages informa tion to minimize the noise of localization . Using this method ,The method, the average error decreased to 2cm in x-axis and 3 cm in y-axis.
Particle Filtering
We implementing Particle filtering, one of the most improved methods on localizationto decrease the amount of the localization error. It considers both current vision percept of agent and its previous states and actions. The method needs the worldmodel to predict the effect of actions.
Team Architecture
Our team architecture consisit of communication layer,decision layer and then skill layer That is shown in the figure 1.we are going to explain the roles of our components in pnu Agent in the following paragraphs briefly.
Communication Component
The communication component is responsible for communicating with the 3d soccer server,i.e Receiving message from the server(agent state,vision information about soccer field , football And other information send by server)and sending command to the server (init,beam,drive,kick And so on).
World Model
this class we have some information about all objects that produce and engaged in Our system.these object are two kinds : dynamic and static.the static object includes information About static and fixed objects in the system such as flags , ines and goals . The dynamic objects Contains some information about moving objectssuch as player and ball.
One of the most important parts of our base is its world.it is designed so that Each agent will be able to accsee precise information about all objectsin the simulation,reliably and simply.in fact, the worldmodel provides the agent with all of information that it wants to need Them. the position of all players , ball and their velocities and accelerations are one of the most important Information that saved in the world model .
Skill Layer
The behavior of an agent depends on the individual skillsthis agent can perform. A skill can be regarded as the ability to execute a cetain action. In general, these skills can be divided in To simple skills that correspond to basic actions and more advanced skills that use the simple skills are parts of more complex behavior.The execution of each skill depends on the current state of the world model which is represented by the agent world model.
Formation Manager
This component consists some information about our team formation method For determinig a stragegic position . For example this class includes an algorithm for automata changing the formation of the team in the different situations .
Decision Making and Learning
Some decision are very complex and difficult and hard to program by hand. for example stop to the position to perform the goalshot , dribbling , passing and etc
Machine Learning Provides algorithms that find good solutions to these problems.
The idea behind our approach is to find a value function M(s,a) that describes how desirable a pair of situation and action is. M is a mapping from a state s and an action a to a value in[0.1]. A value close 1 indicates success, a value close to 0 failure.
The value function is estimated usingQ(Landa) algorithm[1.chapter 7] with landa close to 1.The State s consists of the position of the agent , the ball and the opponents. The action a consists of five different relative positions, the decision whether to kick or not and the kick force.
As we restrict the value of Q to the rang [0,1] , we can reinterpret it as the probability of succsss And use logistic regression [2] as function approximmator. This is mathematically equivalent to a Neural net without hidden layers.
Team Strategy and Collaborative Play
For making a collaborative play, we find out some ways ,and one of team is using some different Formation, in which we had assigned different role to each player and it would change in different Situation.We have located four players as defenders and three as forward and three middleplayer Which will join whether defenders or forwards in differentgame situation . In addition to these players we have introduced a developed Goalie which can chage its position depending on relative position of ball and opponent attackers.
Passes
Straight passes between the agents,forces the receiving agent to turn around the ball for correct alignment. Since aligning takes considerable amount of time , we prefer lead passes Between the agents where the receiving agent can easily collide with the ball. Air Kicks are preferred when the selected teammate to pass is at a longer distanc erelative to the active agent.
Dribbling
When there is no teammate available to receive the ball, we use this skill to main Tain ball possession. We use a 2-phase algorithm for finding the optimal dribble direction. In the first phase , We consider all the opponents within a certain area and enumerate all the possible directions in which our agent can dribble.In the second phase , we select the bestdirection by considering our teammates within a limited region. Dribble length is based on the two nearest opponents in the dribble direction.
Prediction Methods
Some prediction methods like FastGoToBall , NearestToBall , NearestTo Opponent , GetNrCycleToPoint , GetNrPlayersInRegion , PredictStateAfterCommand , Line Of Ball.For predicate the line of ball , when the ball is moving , we draw a imaginative line Among the line of ball moving. Then with ball's velocity and acceleratin , the position of Start and line formulations in the mathematic we predicate the position of ball destination. So drive the nearest agent to the position.[6]
Defense
Align position skill takes considerable time to kick the ball in the desired direction. So, in our defense zone , we use air-kick in the direction of the widest angle between The opponents to move the ball away from our goal post.
References
- Stone P , Layerd Learning in Multi_Agent Systems, phD thesis,School of Computer Science , Carnegie Mellon University ,December 1998.
- Sutton, R.S., Barto, A.G,: Reinforcement Learning: An Introduction. MIT Press(1998).
- Hosmer, D.W.,Lemeshow, S,: Applied logistic regression. Wiley, New York (1989)
- A.G.Barto,S.J.Bradtke,and S.P.Singh. Learning to act real –time dynamic programming . Artificial Intelligence, (72):81 138, 1995.
- Patrick Riley. SPADES System for Parallel Agent Discrete Event Simulation User's Guide and Refrence Manual. 2003.
- Kalman, R.E.: A new approach to linear filtering and prediction problems. Trans. ASME, Series D, Journal of Basic Engineeng 82(1960) 35-45
- A.Saffiotti. Fuzzy Logic in Autonnomous Robotics: behavior coordination. Proc. Of the 6 th IEEE Intl. Conf. on Fuzzy System, pp.573-578.Barcelona,Spain, 1997.
- Smith, and Russell, Open DynamicEngine v0.039 User Guide, 2003.