KylinSky3D Soccer Simulation Team Description for RoboCup2012

Chaohong Cai, Yang Liu, Zhengdong Luo, Hong Jiang

Robot Lab, Hohai University Wentian College, Maanshan City, Anhui Province, China; Computer and Information College, Hohai University, Nanjing City, Jiangsu Province, China

Abstract This paper describes some main features of the KylinSky2012 3D soccer simulation team. A new walking engine was introduced, which is based on 3D-LIPM and our former gait planning method. The walking engine plan the motion of inverted pendulum which connects the supporting foot and the center of mass of the whole robot, and use numeric method to calculate the joints angle. A brief introduction of basic features, such as agent architecture, world model and team coordination, are also presented in this work. Finally we made a summary and future research works are addressed in the last part.

1 Introduction

Our team, KylinSky, was formed in April 2010, and successfully attended RoboCup 3D Soccer Simulation competition 2011. Besides, we have participated ChinaOpen 2010 and 2011, and the robot competition of our province in September 2011. As SimSpark[7] is becoming more stable and powerful, it is possible to play a 11vs11 game. This means team strategy will play a very important role in the game, however, a stable and fast walking gait would enhance the coordination efficiency of the whole team. Consequently, we have done a lot of research works on stable and fast walking gait after RoboCup2011.

To get a stable and fast walking gait, we have built a new walking engine based on 3D-LIPM(3D Linear Inverted Pendulum Model)[10] and our former gait planning method[4]. During the supporting phase, the dynamics of robot can be represented by a single inverted pendulum which connects the supporting foot and the center of mass of the whole robot. The 3D-LIPM uses limited knowledge of dynamics, such as location of total center of mass, total angular momentum and so on. For the trajectory of swing foot, we utilized the theory of synthesized sinusoids[2]. The smooth trajectory which constructed by synthesized sinusoids decreases collisions when robot switch supporting foot, and it could ensure consistency and smoothness of walking.

Because of the final goal is to play game with 22 agents, we adopted some approaches of team coordination. Due to the restricted vision of agent, the formation which decide by each agent may be different, we designed a communication protocol to ensure each agent posses the identical formation strategy. Some fundamental works have also been done, these including: expand the world model; and debug some offline motions, for instance, pass ball motion.

This paper introduces the main features and implementation of our team. Section 2 briefly describe our agent architecture. The world model of agent is illustrated in section 3. Section 4 discuss the walking gait planning method. In section 5, we draw conclusions and present directions for future work.

2 Agent Architecture Architecture

We designed the agent architecture based on mixed partition method[5]. The overall structure of our program consists of three layers: connection layer, technique layer, controlling layer. The function of connection layer is keep a network connection between agent and simulation server, i.e., receive messages from and send agent's commands to the server. Technique layer is divided into four parts, this layer contains Basic parser, World model, Basic motions and Actor. The controlling layer consist of Agent and Strategy, Agent part contains the internal state of robot, Strategy part accounts for role assignment, formation updating and so on. The agent architecture of our team is displayed in Fig.1.

3 World Model

World Model includes perceptual information of agent, exact actions of agent need well-handled perceptual information. These direct information are not sufficient for agent strategy, we must do some reasoning works or extract enough potential information from perceptual messages, World Model aims at this subject. The content of World Model containing: static and dynamic information of game, states of field, particular messages of each agent, and predicted information for strategy[6][9].

We divided World Model into two parts: field state and agent state. Field state reflects states of the whole soccer field, it consist of static and dynamic informations. Static informations are unchangeable, including the size of field, coordinates of flags, specifications of server, etc. Dynamic informations keep updating every cycle, including position and velocity of ball, positions of teammates, the current game state, etc. Agent state describes a state of agent, it also contain static and dynamic informations. Parameters of robot model makes up static informations, and joint angle, walking velocity and current strategy comprised the dynamic part of agent state. Fig.2 illustrates the structure of World Model.

4 Walking gait planning

A stable and fast walking gait is critical, especially as the coordination between teammates becoming more and more important. Agents could finish tasks easily if they have a flexible walking gait. As a result of these factors, a new walking engine which is based on 3D-LIPM was built. In the walking engine, to maximize the range that can be passed within a single supporting phase, double supporting phase was ignored; the trajectory of swing foot was planned by synthesized sinusoids.

The coordinate system of our walking engine comprise two coordinate frames, the pendulum frame S and swing foot frame F. The coordinate frames allow step planning and leg control to be expressed in their natural frame of reference. This ensures that the system stays manageable because each component only acts on a limited amount of information anchored to its appropriate coordinate frame. The pendulum frame maintain the motion of center of mass, it is always offset by $S_{offset}$ towards the supporting foot, the motion of swing foot corresponds to swing foot frame. Fig.3 shows the coordinate system.

4.1 Motion of Center of Mass

The motion of center of mass is based on inverted pendulum which is made up of a point mass and a massless telescopic leg, and it's motion is constrained to move along an arbitrarily defined plane. The model allows a separate controller design for the sagittal and the lateral motion, thus it simplifies a walking pattern generation a great deal. The position and velocity of the center of mass relative to the origin of the inverted pendulum are given by

$$x(t) = x_0 \cdot \cosh(k \cdot t) + x_0 \cdot \frac{1}{k} \cdot \sinh(k \cdot t)$$ (1)

$$x(t) = x_0 \cdot k \cdot \sinh(k \cdot t) + x_0 \cdot \cosh(k \cdot t)$$ (2)

where $k = \sqrt{\frac{g}{h}}$ , g is the gravitational acceleration, h is the height of the center of mass above the ground, $x_0$ is the position of the center of mass relative to the origin of the inverted pendulum at t=0, and $x_0$ is the velocity of the center of mass at t=0. The elimination of double supporting phase acquires agent adjust the point in time while switching supporting foot, we adopted the method from literature [1] to realize this adjustment.

4.2 the Trajectory of Swing Foot

To avoid collisions when robot switch supporting foot, the velocity of swing foot should be zero at the moment of leaving and touching the ground. For the avoidance of jerkiness, the acceleration of these two moment should be zero too. According to the above restrictions, we suppose the acceleration of swing foot is:

$$x_r(t) = A\sin(\frac{2\pi}{T_s}t)$$ (3)

where A is coefficient to be computed, $T_s$ is the cycle of single support phase. From the restrictions, we have $x_r(0) = x_r(T_s) = 0$ , this indicates that there is no jerkiness of swing foot. Do an integral on $x_r(t)$ , we get the velocity:

$$\dot{x}_r(t) = \int x_r(t)dt = \int A\sin(\frac{2\pi}{T_c}t)dt \tag{4}$$

i.e. :

$$\dot{x}{r}(t) = -\frac{AT{s}}{2\pi}\cos(\frac{2\pi}{T_{s}}t) + C_{1}$$ (5)

the constant $C_1$ can be calculated by $x_r(0) = 0$ , then the final equation of velocity is:

$$\dot{x}_r(t) = -\frac{AT_s}{2\pi}\cos(\frac{2\pi}{T_s}t) + \frac{AT_s}{2\pi} \tag{6}$$

after another integral on the velocity $x_r(t)$ , the trajectory of swing foot in the X axis direction is:

$$x_r(t) = \frac{AT_s}{4\pi^2} (2\pi t - T_s \sin(\frac{2\pi}{T_s}t)) + C_2$$ (7)

where A and $C_2$ can be determined by $x_r(0) = 0, x_r(T_s) = D_s$ , thus we get the trajectory of swing foot in the X axis direction:

$$x_{r}(t) = \frac{D_{s}}{T_{s}}t - \frac{D_{s}}{2\pi}\sin(\frac{2\pi}{T_{s}}t)$$ (8)

where $0 \le t \le T_s$ , $D_s$ is the step length of one step

The planning method for swing foot in the Z axis is similar to the process of X axis, but the restrictions are different. The following equations are trajectories of swing foot in the Z axis direction.

$$z_{r}(t) = \frac{2(H_{s} - h_{foot})}{T_{s}} t - \frac{H_{s} - h_{foot}}{2\pi} \sin(\frac{4\pi}{T_{s}}t) + h_{foot} , \quad 0 \le t \le T_{s}/2$$ (9)

$$z_{r}(t) = -\frac{2(H_{s} - h_{foot})}{T_{s}}t + \frac{H_{s} - h_{foot}}{2\pi}\sin(\frac{4\pi}{T_{s}}t) + 2H_{s} - h_{foot}, T_{s} / 2 \le t \le T_{s}$$ (10)

where $H_s$ is the max step height of swing foot, $h_{foot}$ is the height of ankle.

4.3 Ongoing OngoingResearch Research

The inverted pendulum controls the motion of center of mass by limited knowledge of dynamics, since the controller knows little about the system structure, stable walking much relies on a feedback control. An approach based on control theory has been proposed by Kajita et al. [11] in which the CoG is designed to converge at the end of previewing period. Due to the inner tracking error of the preview controller, the magnitude of the preview gain converge slowly[3]. Except the control theory method, reinforcement learning can serve as an alternative method. Reinforcement learning is a learning algorithm that mimics the human learning procedure form experience[8]. Thus, the parameter selection problem will be solved by learning while it is hard for hand tuning method, and the optimal walking gait can be learned by this approach. In our ongoing research work, we will optimize the walking gait by reinforcement learning. The learning procedure will be divided into several layers, different layer take different learning task, the final step is integrate each layer by learning.

5 Team Coordination Coordination

Multi-agent domains consisting of teams of agents that need to collaborate in an adversarial environment. In3D soccer simulation, the increasing number of agents form one team offers great challenge to research works.

During the game, each agent maintains a formation of the whole team. The position of ball and the state of game decides formation of team, each role in formation is calculated by constant offset correspond to the positon of ball. Because of the restricted vison, agents may maintain formation different form each other. We give each agent a specific time slice to communicate with each other, in this way, each agent can share the formation with its teammates, and chose the formation with the most votes. Beside the formation, agent can broadcast its position and ball position to its teammates.

The further work is planned to design a more reasonable coordination system, especially improve the accuracy of role assignment algorithm, and to develope an extensible team strategy with more pass ball actions.

6 Conclusion Conclusionand Future Work

This paper introduced the main features and implementation of our team. The ways we implemented the agent architecture, world model, walking gait planning and team coordination are addressed. The walking gait planning method is based on inverted pendulum, and combines the advantages of our previous gait planning approach.

In our future work, we will use reinforcement learning as an alternative method to optimize the walking gait. Its a learning algorithm that mimics the human learning procedure form experience, the algorithm can learn an optimal policy without ever being able to predict the effects of its actions, and it can solve a lot of problems caused by hand tuning method. Another important part of our future work is design an extensible and flexible coordination strategy. Currently, our team coordination strategy is mostly based on single agent's evaluation, a coordination model rely on single agent's and team's evaluation will be our further plan.

References

1. Colin Graf, Thomas Rofer: A Closed-loop 3D-LIPM Gait for the RoboCup Standard Platform League Humanoid. Proceedings of the 5th Workshop on Humanoid Soccer Robots, Nashville (USA), 2010, December 7, pp.18-22
2. Deng Xin: Stable walking gait of biped walking robot. Master's thesis, Haerbin University of Engineering, 2007
3. Feng Xue, Xiaoping Chen, Jinsu Liu and Daniele Nardi: Real Time Biped Walking Gait Pattern Generator for a Real Robot. RoboCup Symposium 2011, Istanbul, 2011
4. Hong Jiang, Chaohong Cai: KylinSky3D Soccer Simulation Team Description for RoboCup 2011. In RoboCup 2011, Istanbul (Turkey), July, 2011
5. Jiang Chunlu: A research of agent's basic motions based on RoboCup3D simulation. Bachelor's thesis, Southeast University, 2006
6. Liu Rujia, Sun Zengqi: The maintenance of world model for RoboCup3D simulation. Thesis Collection of China Soccer Robot Game, 2005:21-25
7. Markus Rollman: Spark- a generic simulator. Diploma thesis, University Koblenz-Landau, 2004
8. Richard S. Sutton and Andrew G. Barto: Reinforcement Learning: An Introduction. The MIT Press, 1998.
9. Song Beibei: World model design for RoboCup3D simulation. Master's thesis, Hefei University of Industry, 2007
10. Shuuji Kajita, Fumio Kanehiro, Kenji Kaneko, Kazuhito Yokoi and Hirohisa: The 3D Linear Inverted Pendulum Model: A simple modeling for a biped walking pattern generation. Proc. of the 2001 IEEE/RSJ, Hawaii, USA, Oct. 29-Nov. 03, 2001
11. Shuuji Kajita, Fumio Kanehiro, Kenji Kaneko, Kiyoshi Fujiwara, Kensuke Harada, Kazuhito Yokoi and Hirohisa Hirukawa: Biped Walking Pattern Generation by using Preview Control of Zero-Moment Point. Proc. of the 2003 IEEE International Conference on Robotics and Automation, Taipei, Taiwan, September, pp.14-19, 2003