HfutEngine3D Soccer Simulation Team Description Paper 2019

Yincan Wang, Zhuolun Li, Kai Wang, Xinpeng Hu, Hao Wang, Baofu Fang

School of Computer and Information Hefei University of Technology, China

Abstract This paper simply describes the architecture of HfutEngine3D team. This year we use the released code form UT and add ourselves decision information. Additionally, we use CMA-ES method and closed-loop control method of an omnidirectional walking model of humanoid robots which based on Cerebellar Mode Articulation Controller (CMAC) to achieve a steady and fast omnidirectional walk.

1 Introduction

In RoboCup China 2005, we came into RoboCup 3D Simulation League for the first time. Early 3D league was sphere form and we focused on the accuracy of calculation of three-dimensional physical virtual circumstance. At the same time, research of the Middleware SPADES (System for Parallel Agent Discrete Agent Simulation) was also very important. We get 10th place in RoboCup China 2005, 12th place in 2006. In 2007s, new version of server was released, which included new Fujitsu HOAP-2 simulation robot instead of old sphere robot. The new server brought many changes as well as new challenges such as, Joint Control, State Detect and etc.. After months hardwork, our team featured some new controlling ideas and its humanoid motion worked very well. HfutEngine3D got 7th in RoboCup China Open 2007 in Oct.. We also got 3rd in RoboCup Iran Open 2008 and 5th in RoboCup China Open 2008. In July 2008, we attended in RoboCup 2008 in Suzhou. It was the first time for us attended in RoboCup. We have advanced into the top 16. Later we got the 3th place in 2010 in RoboCup,and the 6th in 2012. And we got the 5th in RoboCup China Open 2017. In 2018 we reached 2rd place in RoboCup China Open. This paper introduces HfutEngine3Ds features and implementation.

Section 2 briefly describes the teams main modules. Section 3 introduces some of our team's characteristic. Section 4 tries to show our process based on Cerebellar Mode Articulation Controller to achieve a steady and fast omnidirectional walk. The last section is about our plan of future work.

2 Team Architecture

According to Peter Stones layer learning method, we design four learning modules for the team. They are InformationHandle, MotionHandle, WorldmodelHandle and StrategyHandle.

The InformationHandle takes charge of communication with server, itincludes network controlling, message parsing and command queue building. WorldmodelHandle contains several states updating and some calculating of motions key parameters like whether robot is falling down. MotionHandle is designed to control Joints to finish quite complex task. StrategyHandle is the brain of robot which is based on non-goalie idea and dynamic assignment.

As 3D server is working in C/S mode, we get message from InformationHandle module firstly. After parsing the message, WorldmodelHandle module updates all of the states including joints state, game state and object state(ball, itself, teammates, op-ponents and etc.). StrategyHandle module analyzes current situation and then chooses one tactic with the best benefit. To achieve the strategy, the robot should also make a series of joints commands to perform motions finished by MotionHandle module,joints commands will be put into the command queue. Atlast, InformationHandle module gets command from command queue.

3 Team Characteristic

(Section containing subsections below)

3.1 Self-Localization

We only need three position vectors about the relationship between flags and our robot to calculate its self-position. The robots x-coordinate will be calculated by length-ways and two flags, while the y-coordinate will be given by transverse two flags (It is known that any three flags of field s arrange have two ones in lengthways and two ones in transverse, so we choose the closest three flags generally).

In Fig.2, parameters d1, d2 and d3 denote the distance between robot and Flag1, Flag2 and Flag3, which are included by vision information. We can draw a triangle with field width, d1 and d2 in level. With the triangle, we can calculate the ycoordinate. Moreover, we get the x-coordinate by another triangle with field length, d1, d3. At last, we use the x-coordinate and the y-coordinate to get robots height. It can also be obtained by forward kinematics when the robot is standing.

3.2 Filtering

In the early versions of server, client can get completely accurate visual information,so we can get accurate coordinates. While after some updates,the noise jamming is added. The visual information got from server is no longeraccurate.In order to reduce the error and get the accurate coordinates of position, we use Kalman filtering algorithm to process the data.Comparing the coordinates before and after filtering, we believe that the filtering is effective.

3.3 Kinematics

The detailed parameters ofthe robot are quite important to the robot's development. With these parameters we can construct forward kinematics and inverse kinematics. In our team, forward kinematics is used to get the position of joints, even the height of robot. While inverse kinematics is often used to calculate the joint angle after walking gait planning.

3.4 3D Linear Inverted Pendulum

In the latest version of HfutEngine, we use 3D linear inverted pendulum to plan the Walking pattern. Itcan make robots walking faster and more stable without falling down too often. The following equation is the key to the planning of walking pattern:

$$\bar{x} = \frac{g}{z_0} x$$

The equation of y-coordinate is the same as the one below. We can use this equation to calculate the x-coordinate and y-coordinate of the torso, moving foot and holding foot as the time changes. Then, use inverse kinematics functions to get the angle of each joint, and then apply these angles that have calculated, the robot can be able to walk more perfectly.

3.5 Shift Velocity Walking

Sometimes, our robots may directly walk to a random point. What we need to do is just controlling the accelerating and decelerating work period. The following is the characteristics ofwalking what we want our robot to make:

1. Dynamicly calculate the acceleration based on the distance to target andangle.
2. Detect the change of target, dynamicly switch different action.
3. Predict possible stance in all walking period including walking distance,velocity and the slowdown point.

To smoothing the shift process, we construct a logarithm function which is based on distance.

$$a = 1.67607lq(DistanceToTarget)$$

When calculating the slowdown point, we need to predict the max velocity the robot can reach. After analyzing from experimental data, we use a quadratic function to pre-dict the max velocity. Here are three groups ofdata:(d1,Vm1), (d2,Vm2), (d3,Vm3).

$$V_m = V_{m1} + \frac{V_{m2} + V_{m1}}{d_2 - d_1} (d - d_1) + \frac{\frac{V_{m3} - V_{m2}}{d_3 - d_2} - \frac{V_{m2} - V_{m1}}{d_2 - d_1}}{d_2 - d_1} (d - d_1) (d - d_2)$$

3.6 Actions

We add some actions in the latest version, such as kicking balls from the front or side. With the help of 3D Linear Inverted Pendulum, robots can make some complex actions. To cooperate with these actions, we also add some simple Articial Intelligence to help robots determine where and when to do the actions.

As an example, in Fig.3, we can see in the previous version, when robot(the big R in Fig.3) gets the ball's position and the target position, it adjusts its facing angle then walks directly to the ball. In our recent version, robot can get the path curve to the ballwith its facing angle, the ball's position and the target position. That will save time to get to the target and increase the accuracy of shoots.

3.7 Decisions

We make some functions for robots to decide theirnext movements. These functions are divided into two categories, individual decisions and team decisions. Robot relies on its self-location and the ball's position to decide what individual decision it takes.

Team decisions are used to controlformation transform and other movements. To sum up, team decisions govern the formation, and individual decisions decides the speci al movement of individual robots.

3.8 Research on Gait Parameters Optimization

In order to achieve a fast and stable gait, we did it with a humanoid robot helix model algorithm based on the covariance matrix adaptive evolution strategy (CMA-ES). In the walking optimization process, the optimization task is first divided into three sub areas. Each optimization group's CMA-ES algorithm optimizer is designed according to different learning objectives. Each optimization group's learning is combined with the new requirements based on the optimization results of the previous optimization group to perform spiral iterative optimization, and ultimately achieve the established learning goals. In order to obtain the best parameter value. This method uses the NAO robot model as an object and applies it to the HfutEngine simulation 3D team. The robot's related gait test data shows that this method has better results.

3.9 Toolkits Development

We developed a Controller to control the rotation angle of robots' joint. It's easy for user to use sliders and spins to control the angles ofjoints. By using this controller, we can simulate the status of robot in motion. The tool is connected to the simulation server through a TCP socket. After change the sliders' position and spins' status, clicking the 'send' button, robot's joints would rotate to the angle that we set before. It was developed by Qt 5, and it can be executed in Linux.

4 Experiments in CMAC

This year we use a closed-loop control method of an omnidirectional walking model of humanoid robots which based on Cerebellar Mode Articulation Controller (CMAC).The cubic spline interpolation method were used to plan the foot space trajec-tory, and built the Double Linear Pendulum (D-LIP). Subsequently, the trunk and foot poses were applied to compute the currentjoint angle value of the robot legs. Mean-while, the sensor information according to the CMAC model can be modified the angle value in the Inverse Kinematics to achieve a steady and fast omnidirectional walk.

5 Future Works

At present, we are doing some research about using DDPG-algorithm(Deep Deterministic Policy Gradient) to make humanoid robot learning how to walk by itself. We have successfully applied for National innovation and Entrepreneurship Project about that research. By far, we are working on this research and it's progressing smoothly.

After that, we plan to keep working on the optimization of Long-Distance Kicking and strategy optimazing. And we still want to develop our tools for the robot debug and design a good algorithm for planningthe cooperation of robots. We still have a long-range goal to achieve.

References

1. Kajita, S. and Tani, K.. Experimental study of biped dynamic walking. In IEEE Control Systems, Vol.16, No.1, pp. 13-19, February 1996.
2. Kajita, S.. Humanoid Robots. Tsinghua University Press, 2007.
3. R.Tedrake, T.W.Zhang, H.S.Seung. Stochastic Policy Gradient ReinforcementLearning on a Simple 3D Biped. In Proceedings of 2004 IEEE/RSJ International Conference on Intelli gent Robots and Systems (IROS2004), pp. 2849-2854, 2004.
4. Guestrin, C., Venkataraman, S., Koller, D.. Context-specific multi-agent coordination and planning with factored MDPs. In Proc. 8th Nation. Conf. on Artificial Intelligence, Edmon-ton, Canada (2002).
5. Kok, J.R., Spaan, M.T.J., Vlassis, N.. Non-communicative multi-robot coordinationin dynamic environments. In Robotics and Autonomous Systems 50 (2005), pp. 99, 114.
6. Kajita, S. ; Kanehiro, F. ; Kaneko, K. ; Yokoi, K. ; Hirukawa, H. The 3D linearinverted pendulum mode: a simple modeling for a biped walking pattern generation.In Intelligent Robots and Systems, 2001. Proceedings. 2001 IEEE/RSJ International Conference on, pp. 239 .
7. Zhulin An, Jiangjiang Yu, Hao Wang. RoboCup Simulation League Goalie Design.In Proceedings of1st Austria Open of RoboCup, 2003.
8. Baofu Fang, Hao Wang, Jia Liu, Chenwen Su.Team Strategy of HfutAgent. In Proceedings of2003 Master China RoboCup, Aug, 2003.
9. Cheng Wang, Hao Wang, Baofu Fang. Multi-agents Action Selection Using Coordination Graph Based on Value Rule. In Computer Engineering and Applications, 2004(19).
10. Lei Yu, Hao Wang, Cheng Wang. Studies on Strategy of Pass in RoboCup. InComputer Engineering and Applications, 2004(8), Aug, 2004
11. Baofu Fang, Hao Wang, Hongliang Yao, Jin Yang, Jin Zhou. The Application ofQ Learning In RoboCup. In Proceedings of2004 China RoboCup, Oct, 2004.
12. Runmei Zhang, Hao Wang, Honghang Yao, Baofu Fang. Influence Diagrams andIts Appli cation in Robocup. In Acta Simulata Systematica Sinica, 2005(1), 2005.
13. Baofu Fang, Hao Wang. The Survey of HfutEngine2005 Robot Simulation SoccerTeam De-sign. In Journal of Hefei University of Technology, Vol.29(9), Sep 2006.
14. Jianqing Gao, Hao Wang, Lei Yu. A Fuzzy Reinforcement Learning Algorithm andIts Application in RoboCup Environment. In Computer Engineering and Applications, 2006(6).
15. Yang Liu, Hao Wang, Baofu Fang, Hongliang Yao. Application of the method of support vector regression in RoboCup. In Journal of Hefei University of Technology, Vol.30(10), Oct 2007.
16. Baofu Fang, Bingrong Hong, Hao Wang, Dunqiao Bao, Long Li. A Muti-AgentDefensive Strategy Based on Monte Carlo Method. In Journal of Harbin University of Technology, 39(S1), Jun 2007.
17. Hao Wang, Yang Liu, Baofu Fang. The Research of Reinforcement LearningTechnical Based on Support Vector Machine Classification and Its Application in RoboCup. In Journal of Harbin University of Technology. 39(S1), Jun 2007
18. Mike Depinet, Patrick MacAlpine, and Peter Stone. Keyframe Sampling, Optimization, and Behavior Integration: Toward Long-Distance Kicking in the RoboCup 3D Simulation League. Proceeding of the RoboCup International Symposium 2014, July 2014
19. Hansen N. The CMA evolution strategy: A tutorial[J]. arXiv preprint arXiv:1604.00772, 2016.
20. Hansen N. The CMA Evolution Strategy:A Comparing Review[M]// Towards a New Evolutionary Computation. Springer Berlin Heidelberg, 2006:75-102.
21. Urieli D, Macalpine P, Kalyanakrishnan S, et al. On optimizing interdependent skills: A case study in simulated 3D humanoid robot soccer[C]// International Conference on Autonomous Agents and Multiagent Systems. DBLP, 2011:769-776.
22. Macalpine P, Stone P. Overlapping layered learning ☆[J]. Artificial Intelligence, 2018, 254:21-43.
23. 薛建. 基于 CMA-ES 算法的足球仿人机器人步态研究与实现[D]. 合 肥工业大学, 2014.
24. Zhou Z, Zhou Y,Wang J.Omnidirectional walk design of humanoid robots using layered learning method based on CMA-ES[C]// Advanced Information Management, Communicates, Electronic and Automation Control Conference. IEEE, 2017:464-468.