HfutEngine2022 Simulation2D Team Description Pape

Yansong Yu, Gongfang Zhao, Hao Wang, BaoFu Fang

https://www.robocup.org/

Abstract This paper mainly describes the related work of HfutEngine2D in 2021 -2022, including using genetic algorithms to adjust parameters and trying to use Neural Networks to make more rational decisions. Combined with the characteristics of HfutEngine2D, this paper expounds the al-gorithm concept, design intention, application process and progress in detail. In our recent test with other RoboCup strong teams, the strength of HfutEngine2D team has been sig nificantly improved, and for the long-term development of the team, these im provements have also played a great benefit.

1 Introduction

HfutEngine2D was founded in 2002 and began to participate in the China robot competition in the same year. In the next few years, HfutEngine2D developed rapidly and participated in various competitions. Since 2003, we have been using agent-2d as our bottom code, with the update of server version, we use BP neural network algorithm in intercepting learning, value-based learning algorithm in action selection, reinforcement learning algorithm and a series of machine learning algorithm in passing action learning, which has achieved good results. We have won the runner up of 2007 RoboCup China open, the seventh place of 2008 World Cup RoboCup, the bronze medal of 2008 Iran open, the fourth place of 2009 RoboCup China Open Simulation Group 2D, the champion of 2009 Iran open, and the simulation group 2D of 2011 RoboCup China open.We use agent2d as our underlying code to build HfutEngine2D in 2014.I n the 2015 World Cup, we finished eighth in the simulation 2D group. We also participated in the 2016 and 2017 World Cup. In recent years, we won the championship in the RoboCup robot world cup in China in 2019, the 10th place in the RoboCup robot world cup in 2019, and the runner up in the RoboCup robot world cup in China in 2021.

2 model and approach

2.1 genetic algorithms

In our robot team, there are tens of thousands of parameters that need to be adjusted, because of the large number of parameters, the "brute force method" is expensive. In this case, genetic algorithm is t he best choice. The selection of criterion, the setting of variation and the initial value are all very important in genetic algorithm, otherwise it is easy to be trapped in local optimization.

In HfutEngine 2022, we divide the modification targets into three categories :1. Whether a piece of functional code exists; 2. Parametric data. Then we put all the parameters in a set of code as an individual, a group of such an individual as a species, and they have different genes, we put on a different team winning percentage as fitness function evaluation standard, first record the match in each iteration and the corresponding parameters, and then according to the fitness eliminate a certain number of teams, And let all individuals produce a certain amount of variation, in the process of continuous iteration, we gquet better individuals.

The genetic algorithm uses 10 individuals as a generation, and uses the 100- game winning rate for different teams as the fitness to eliminate underperforming individuals, and adjusts the individual differences through simple random up and down. We use about 50 generations of variation. It is found that the win rate peaks, and then the win rate stabilizes around this value.

2.2 neural network

We once used the deep learning method to optimize the action and decisions of a single player, but the overall decision. We hope to evaluate the scoring function of high-level actions (passing, dribbling, shooting) through the neural network, learn strategy, do not improve personal behavior, but improve team strategy.

To get a better strategy, we let the stronger teams play with us, get the log data, record the position of the positive and negative action ball before the strong teams, find the offensive and defensive positions of the strong teams, and thus imitate the strategy of the strong teams We trained the neural network through supervised learning, setting positive and negative events with a target value of 1 / 0.

We built neural networks using the pytorch framework,The input layer is the position of the next kick, and the fragments extracted from the log are converted to the training data, with the input features (x, y) and positive and negative target values.The output layer is the probability of a sigmoid function —— entering the opponent's penalty area, assuming that team members will consider team strategy to approach experts by modelling each player's action selector as an expert action selector.Take the ball into the penalty area and have a low action score with low success probability.The neural network structure is shown in the Figure 2.

We import the libtorch library at compile time by rewriting the cmakelist to deploy the pytorchtrained model. Finally, the results of neural network evaluation and our already existing evaluation functions combine the strategy of optimizing the overall system by changing the estimator.

3 partial optimization

3.1 Tackle optimization based on tackle rate

We take the probability of tackle success and the probability of foul as input vectors, and the values of T and F of the input vectors are output online by the player agent and saved to a file. Saves the period at which the tackle action is performed, the defensive player number, the enemy offensive player number, the T value, and the F value. The cycle and offensive player numbers are used to quickly locate player-related information when analyzing Rcg files. According to whether the player performs a dash atomic action after being shoveled, the tackle action is labeled. In the action information in Rcg, there is a statistical number of atomic action dashes dash count. If the tackle fails, the shoveled player will generally continue to carry the ball forward, and if the tackle is successful, the shoveled player will not immediately perform the dash action.

Train the model using the logistic regression method to decide whether to tackle or not. The network framework is pytorch, the model structure is shown in the Figure 3, the model deployment mode is the same as above, the number of iterations T = 1500, and the learning rate is α=0.03.

3.2 Defensive optimization

When the opposing player enters our half, the most important thing is to intercept the passing ball of the opposing team and disrupt the passing strategy of the opposing team. In order to improve the success rate of interception, it is a key step to choose the interception position in advance.

• 1.The position of the goalkeeper was moved back according to the position of the ball.
• 2.Added condition judgment and position selection for defenders to intercept shots in front of goal. According to the characteristics of the attack, the distribution of the ball, goalkeeper position selection, and scoring.
• 3.Added pass-block interception mode, which predicts the next possible receiving opponent based on the distribution of the opponent holding the ball and other opponents, and selects the interception position and scores accordingly.
• 4.According to the position selection of ball-block interception mode, the position of interception is predicted and scored by the state of the ball. Conclusion: Increasing the more likely positions and reducing the less likely positions through judgment and conditional screening, and increasing the success rate of interception after scoring. Future direction: Optimize the evaluation model to increase the success rate of ball interception.

4 Experiment

Conducting 50 game tests to count the number of tackle successes and failures of Full Back , This is shown in the chart.the tackle failure rate dropped from 7.39% to 5.87%, indicating that the full-back's new tackle strategy had some effect, reducing the possibility of tackle failure.

Tackle Results

we collected the deployment of HFUTengine2022 after the genetic algorithm, the winning rate of HFUTengine2022 for each domestic team, after 200 tests, the test results are as shown in the chart, it can be seen that after the genetic algorithm is adjusted, the winning rate of each team has improved.

Team Test Results

5 Conclusions

HfutEngine2D has undergone some optimizations based on neural networks and genetic algorithms. We briefly describe the basic principles and implement them according to their own characteristics. At the same time, we have done some work on the optimization of the strategy and the optimization of the power model. HfutEngine2D is now a big improvement over the previous version. Tackle success rates have improved and attacks in the penalty area are more efficient. In the next phase, we will try to train the model to analyze the opponent's offense to improve defensive ability, and optimize the current imitation learning model to construct a search tree so that it can achieve multi-step evaluation.

References

1. Hidehisa Akiyama and Tomoharu Nakashima. Helios base: An open source package for the robocup soccer 2d simulation. In Sven Behnke, Manuela Veloso, Arnoud Visser, and Rong Xiong, editors, RoboCup 2013: Robot World Cup XVII, pages 528–535, Berlin, Heidelberg, 2014. Springer Berlin Heidelberg.
2. Fang Baofu Robot soccer simulation [M] Hefei University of Technology Press, 2011
3. Wang Junyang, Cao Mingkuan, Wang Hao and Fang BaofuHfutEngine2021 Team Description Paper2021
4. Li long Value based machine learning method and its application in RoboCup Simulation 2D [D] Hefei University of technology, 2009
5. Mimicking an Expert Team Through the Learning of Evaluation Functions from Action Sequences[C]// Takuya Fukushima, Tomoharu Nakashima, Hidehisa Akiyama,2018
6. Zhou ZhiHua.Machine learning[M].tsinghua university press, 2016.
7. Peter Harrington. Machine learning practice[M]. People's Posts and Telecommunications Publishing House, 2013.
8. Toby Segaran. Collective Intelligence Programming [M]. Publishing House of Electronics Industry, 2015.
9. Ian Goodfellow, Yoshua Bengio. Deep Learning [M]. Posts and Telecommunications Press.2017
10. Zhuang Yanbin, LI Chunguang, Yang Xiaoyun. Research on the new Method of RoboCup Robot Tackle based on BP neural Network
11. Chen rongya Research on player cooperation strategy in simulated robot football [D] University of science and technology of China, 2015
12. RoboCup Official Site, https://www.robocup.org/.