Alzahra Soccer 3D Simulation Team Team Description Paper for RoboCup 2011
Seyedeh Atieh Hashemzadeh, Mahsa Darab, Seyedeh Zahra Mortazavi, Mahdieh Ahmadi
Alzahra University Tehran, Iran
Abstract In this paper using a PID controller, new methods for kicking and decision making are presented. The activities were focused to get ready for the RoboCup 2011 competition. The objective of the team is to emerge new methods for simulated robots, in a way that their movements are as similar as possible to human beings'. First, the PID controller is introduced, and then we propose a kick method that would enable the robot to hit the ball in appropriate direction with high standard of stability, and we optimize it by Reinforcement Learning (RL) method. At last, we used a new algorithm about decision making for players' mood.
1 Introduction
The concept of this paper is introduced in the following sections:
In section 2, we describe the PID controller that is used to decrease errors in sending angles. Section 3 shows the Inverse Kinematic for different actions such as turn, kick and etc. They are calculated by joint angles.
In section 4, different actions are described. The actions include walking and kicking, that the kick action is divided to five step, and four phases. Also to reach the ball we proposed the Reinforcement Learning method that searches suitable area to put its foot there.
In section 5, an algorithm for decision making is presented. We assumed either Defensive or Offensive mood for each player. At the beginning of every cycle, we consider the conditions. In addition, controlling the angle of robots, well behavior of robot, hitting of ball in different directions and making decisions are among the ideas for Alzahra team. This team has participated in the following competitions: 2nd National Khwarizmi Robotic Competition, Iran Open 2010, German Open 2010, RoboCup 2010 in Singapore and PNU Open 2011. Also we have invited for taking part in German Open 2011.
Notice that, our code is based on Zigorat-0.6.1 base code.
2 PID Controller
One of our troubles in the team was to send angles with high angular velocity for some actions which caused in terrible performance of our robots. In other words by plotting the inputs (sent angles to server) and outputs (current robot's angles) we have got a lot of errors.
A suggested solution to this problem was to use a controller, so we used a PID controller here with the following formula:
$$a(t) = k_p [e(t) + \left(\frac{1}{K_i}\right) \int_{0}^{t_0} e(t) . d(t) + K_d(\frac{de(t)}{d(t)})]$$
In which:
t is Time, e(t) is error: the difference between the sent and current angles, Kp , Ki , Kd are constants, a(t) is the fitted angle to be sent to server, And Kp(gain), Ki , Kd are constants have been calculated using the Ziegler-Nichols method[1].
About e(t), since our goal is to send angles with different gains which is in contrast to the assumption that Kp is constant, therefore to do this, first we multiply the gain into a given angle and then difference between the obtained and the current angle will be assigned to e(t) as an input.
Another thing to take into account for e(t) assignment is the delay between the phase of receiving angles in sever and running the command.
Eventually to integrate and differentiate it is fair enough to utilize the computational methods. Not to be forgotten that one of the drawbacks of this method is to have a low speed for running some actions.
3 Inverse Kinematic
Based on the gained experiences, to run the actions such as kick, turn, stand up ... we used the Inverse Kinematic method in which by setting a given foot location, the joints angles will be calculated[2].
4 Actions:
(Main section containing subsections 4.1 and 4.2)
4.1 Walking
As we have discussed, our goal was to work on new scientific issues, on the other hand to compete better it is needed to be at the same level of world quality teams, so this year we utilized the Walking procedure of Apollo's team with minor changes in our code.
4.2 Kick
Normally the Kick action is grouped to some categories: Instep kick, Pike kick, Lob kick, Drop kick, Inside edge pass.
Our team works on Instep Kick now and working on other categories is the future works.
Kick action is basically grouped into following phases: Preparation, Retraction, Execution and Wrap up; that in every phase, the joints' locations will be calculated through the Inverse Kinematic (IK) method which has been already described. Actually the suitable kick is a kick that the robot by any angle, distance or position from the ball could be able to hit this, and finally the ball in the least time get to the target. It is also important that during this action, the robot keep its stability.
To achieve this goal, we use one of the famous methods i.e. Reinforcement Learning (RL). In this method we define the points for the robot, and the robot shows special behavior at different points. We test these behaviors and chose the best ones.
Parameters which we want for each of the states are the ball distance from robot (d), the robot relative ball angle (β) and the ball target direction (α). For a proper kicking, we define special points as sates of the robot for RL. In this way, d1,…,di are the distances of ball from robot, β 1,…, β j are the angles of robot from ball, α1,…,αk are the angles of ball to target that i,j,k €N . In general; our domain is divided to m states that m is i×j×k.
5 Decision Making
In the past, we assumed either Defensive or Offensive mood for each player (except for the goal keeper).But this act, causes some disorders in the game process, especially when one or some players with the same mood were out of the game (due to falling down, being sent out due to a fault, losing the position). So by floating the moods between the players we partially sorted this out.
It is done like this: first we assign a mood to a player and with the consideration of the conditions we change these moods between agents, for example, we set a player in Offensive mood and the other in Defensive. At the beginning of every cycle, we consider the conditions as below in the Decision Function:
if (attacker is OUT of the game) $$d_A := \infty && \theta_A := \infty$$ else if (defender is OUT of the game) $$d_D := \infty && \theta_D := \infty$$
$$if((d_D - d_A < Error_{Distance}) && (|\theta_D| - |\theta_A| < Error_{Angle}))$$ Switch the Defender and Attacker moods;
It is assumed that $d_A$ and $d_D$ are the distance from the ball to the attacker and defender, respectively. $\theta_A$ is the angle between the global direction of each attacker and the ball-attacker direction. If defender was considered, the definition refers to $\theta_D[6]$ .
On the other hand, for each mood the following procedure is considered:
References
- http://en.wikipedia.org/wiki/PID_controller
- van Masaki Ogino1, Yutaka Katoh1, Masahiro Aono1, Minoru Asada1*, 2 and Koh osoda1*, 2 1Dept. of Adaptive Machine Systems, 2HANDAI Frontier Research Center, Graduate School of Engineering, Osaka University: Vision-Based Reinforcement Learning for Humanoid Behavior Generation with Rhythmic Walking Parameters. Recent Trends and Developments. Lecture Notes in Computer Science, Vol. 1000. Springer-Verlag, Berlin Heidelberg New York (1995)
- Richard S. Sutton and Andrew G. Barto.: Reinforcement Learning: An Introduction. In: Mark Lee 2005-01-04
- Juan Cristóbal Zagal and Javier Ruiz-del-Solar Department of Electrical Engineering, Universidad de Chile.: Learning to Kick the Ball Using Back to Reality. Av. Tupper 2007, 6513027 Santiago, Chile
- Heinrich Mellmann* and Yuan Xu*Department of Computer Science Artificial Intelligence Laboratory Humboldt-Universität zu Berlin Unter den Linden 6 10099 Berlin, Germany.: Adaptive Motion Control with Visual Feedback for a Humanoid Robot. The 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems October 18-22, 2010, Taipei, Taiwan
- Av. Padre Tomás Pereira S.J., Taipa, Macao SAR, P. R. China.: Multi-Agent Control Structure for a Vision Based Robot Soccer System. 11th IEEE International Conference on Mechatronics and Machine Vision in Practice, Macau, 2004