HfutEngine2021 Team Description Paper

Wang junyang, Cao Mingkuan, Wang Hao, Fang Baofu

Artificial Intelligence and Data Mining Lab School of Computer and information HeFei University of Technology, P.R.China

Abstract This paper mainly describes the related work of HfutEngine2D in 2019-2021, including the optimization of running mode, the optimization of passing and the application of logistic regression model. Combined with the characteristics of HfutEngine2D, this paper expounds the algorithm concept, design intention, application process and progress in detail. In our recent test with other RoboCup strong teams, the strength of HfutEngine2D team has been significantly improved, and for the longterm development of the team, these improvements have also played a great benefit.

1 Team overview

HfutEngine2D was founded in 2002 and began to participate in the China robot competition in the same year. In the next few years, HfutEngine2D developed rapidly and participated in various competitions. Since 2003, we have been using UVA Base2003 as our bottom code, with the update of server version, we use BP neural network algorithm in intercepting learning, value-based learning algorithm in action selection, reinforcement learning algorithm and a series of machine learning algorithm in passing action learning, which has achieved good results. We have won the runner up of 2007 RoboCup China open, the seventh place of 2008 World Cup RoboCup, the bronze medal of 2008 Iran open, the fourth place of 2009 RoboCup China Open Simulation Group 2D, the champion of 2009 Iran open, and the simulation group 2D of 2011 RoboCup China open. We use agent2d as our underlying code to build HfutEngine2D in 2014. In the 2015 World Cup, we finished eighth in the simulation 2D group. We also participated in the 2016 and 2017 World Cup. In recent years, we won the championship in the RoboCup robot world cup in China in 2019, the 10th place in the RoboCup robot world cup in 2019, and the runner up in the RoboCup robot world cup in China in 2021.

2 Running mode

The running mode in HfutEngine is a crucial component for team coordination and positioning.

2.1 existing mode

As a sport of 11 players, football can't win without mutual cooperation, which is also the case on our simulation platform. In our attack, although the decisionmaking of the players with the ball is very important (the rewriting of the action chain is very important), no matter how good the decision-making is without the cooperation of other players and making the corresponding running position without the ball. The corresponding performance of this situation on the court is - we have very few offensive methods, the ball can not run between the offensive players, unable to form a subtle cooperation. And that's the problem we're facing now. In recent versions of the team, we've been improving the running mode.

In HfutEngine 2019-2021, we constantly modify the running mode of the bottom layer, but it is basically based on the following mode: the running mode of the bottom layer is based on the standard point determined by the formation file, and a unique standard point can be obtained according to the distribution of players on the field and the coordinates of the ball. On the basis of the standard point, different additional displacements are introduced to classify the current situation on the field to get the final result The goal is to move the position, so that the formation can be maintained in the game and the players can be evenly distributed.

This mode has a good effect at the beginning, but with the deepening of the improvement, its disadvantages are gradually exposed. When a run is not good, you can only separate this situation and make a separate strategy, but the situation on the court is unlimited. With the refinement of classification, the number of codes will increase greatly. If there is no detailed comments, the readability is extremely low, which brings some problems, the effect is very little. And the basis of this model is that we need to preset an optimal running point. However, we are not professional football players. In many cases, we can not determine the optimal running point or even an area. At this time, we need to continue to explore and supplement this model.

2.2 improvement ideas

In order to solve these problems, we use a new mode to determine the running point, which is based on the scoring mode. First, according to the coordinates of the ball and players to determine the running frame, and then do discrete processing to the running frame, traverse the points in the frame, into the scoring machine, so as to select the best running point. Using this mode can get rid of the trap of constantly refining classification, and it is more flexible, controllable, and the code can be more concise, which is more conducive to the future development.

As shown in (Fig. 3 shows an example3)., at this time, No. 7 of our team holds the ball, and the next running position of No. 11 player is very important. According to the new mode, first determine the running position box, which is the black box on the graph, and discretize the running position box, which is shown as the white point on the graph. Replace all the white points into the scorer, and the one with the highest score is the target running position point. Of course, No. 9 and No. 10 are also doing this Active running without the ball, compared with the previous formation point mode, this mode is easier to find the opponent's defense flaws.

2.3 Application of improvement in exploration

The key of this model is the establishment of scorer. Although every good ball has something in common, the definition of good ball is different for different players and different situations, so the classification is still necessary. However, the classification here only needs to be clear about the major categories, and it does not need to be accurate to the details. However, the workload is still large. At present, we have established a basic scorer, which only aims at the former The front runs without the ball when attacking. After the improvement, we tested several RoboCup teams. Although we have made some improvements, we think it has unlimited potential, especially after the application of machine learning algorithm, there should be more improvements.

comparison of test results

3 Application of logistic regression model in passing

The application of logistic regression in breakthrough passing strategy is a key innovation.

3.1 Question raising

In RoboCup Simulation 2D game, the process of bringing the ball from our half court to the other side's restricted area can best reflect a team's offensive ability and comprehensive strength. Because only when the ball is successfully brought into the penalty area of the opponent's court can we have the chance to break the goal, and keeping the ball in the penalty area of the opponent is also very helpful to enhance our attack threat. Because of this, the designers of all teams will try their best to prevent the other side from bringing the ball into their own restricted area on the defensive side and bring the ball into the other side's restricted area on the offensive side when designing team strategies.

In our previous design version, the usual defensive strategy to face this problem is to intercept or tackle the ball to prevent the opponent from entering the penalty area. It is difficult to find an effective attack strategy. The attacking party should try its best to break through the defense and transport the ball to the other side's restricted area. This is a very difficult thing, but once it is reached, the dividend is huge. In the team version of HfutEngine2017, we try to bring the ball into the opponent's penalty area by designing the pass strategy for a specific area of the course. In developing HfutEngine 2019, we try to improve the scoring of specific areas in the course in the evaluator to keep the ball in the enemy's restricted area as much as possible.

In last year's HfutEngine2020 version, we tried to use the formation editor to fine adjust the running position in the game, so that our players can bring the ball into the enemy's restricted area as much as possible. However, in the actual video observation of the game, we found that often some can penetrate the enemy formation and pass the ball of several enemy players at one time (we call it "breakthrough pass") is the main reason to bring the ball near the enemy's restricted area. In a game, it is often such several opportunities that lead to goals, even when there are multiple breakthroughs The percentage of winning games is very high. This makes us have to start to explore whether there is a connection between breakthrough passing and goal difference, and between breakthrough passing and victory in the game, and how to improve the proportion of breakthrough passing in the game.

3.2 further research on breakthrough passing

Through the analysis of the game log, we can find out which passes are breakthrough passes. We use regular expressions to process the log information, and then use some criteria to determine whether each pass is a breakthrough pass. Then, in order to explore whether there is a relationship between breakthrough passing and the number of goal difference, we analyze the correlation between the number of net breakthroughs and the number of goal difference, that is, to obtain the Pearson correlation coefficient of the two. In 100 games randomly selected, the scatter diagram of breakthrough pass and goal difference is as follows:

After calculation, the correlation coefficient between them is 0.7654. When looking for a breakthrough pass, we only consider the situation of midfield breakthrough, and whether we can score after breakthrough is also affected by many factors, such as the offensive strategy of the attacker and the defensive strategy of the defender in the restricted area, and the defensive strength of the goal-keeper all affect the final result, which explains why the correlation coefficient has a certain gap from 1. However, it can be seen from the existing results that there is a great positive correlation between the number of breakthrough passes and the number of goal difference.

3.3 logistic transformation and logistic regression model

When we establish the regression equation, the value range of the dependent variable is the real number set; when we feel whether to pass the ball, the value of these dependent variables is between 0—1, such as 0.1, 0.5, 0.8, etc., so we need to do logit transformation on the value of the dependent variable (target probability).

Let p be the probability of occurrence of an event and 1-p be the probability of non occurrence, then $\frac{p}{1-p}$ is called the occurrence ratio of an event, which is recorded as odd

$$logit(p) = \ln\left(\frac{p}{1-p}\right) \tag{1}$$

Obviously, when p=1, logit(p) is $+\infty$ ; when p=0.5, logit(p)=0, when p=0, logit(p) is $-\infty$ . In this way, the value range of dependent variable is extended from $0 \sim 1$ to real number set, and the regression analysis using this method is logistic regression.

There is a binary dependent variable y, when the value is 1, it means that the event happened, when the value is 0, it means that the event did not happen; the dependent variable has m influencing factors (independent variables): $x_1, x_2, \dots, x_m$ . The conditional probability $P(y=1|x_i)=p_i$ . Then the logistic regression model of $p_i$ (the ith observation) was as follows

$$logit(p_i) = \ln\left(\frac{p_i}{1 - p_i}\right) = \beta_0 + \beta_1 x_1 + \dots + \beta_m x_m = \beta_0 + \sum_{j=1}^m \beta_j x_j, j = 1, 2, \dots, m$$ (2)

Where $\beta_j$ is the independent variable, when $x_j$ changes a unit, the change of $logit(p_i)$ can be understood as the weight coefficient of each influencing factor. Through transformation, logistic regression model can also be written as follows:

$$p_i = \frac{e^{\beta_0 + \sum_{j=1}^m \beta_j x_j}}{1 + e^{\beta_0 + \sum_{j=1}^m \beta_j x_j}}$$ (3)

3.4 application and result exploration

We first extract the position and speed of players and balls in each cycle from the log. And according to the direction of the pass, the number of passes in the pass and other standards, judge whether the pass is a breakthrough pass. After that, take the data of breakthrough pass as positive example, and those not judged as breakthrough pass as counter example. Use the position and speed of the current team member, the passing team member and the opponent on the pass route as input, and whether it is breakthrough pass as output to train the logistic regression model. After training, the accuracy of the model is 87.34

Then, when applying the model, we modify the relevant part of the evaluator in the team source code, modify the score of the breakthrough pass, and if the current pass is predicted to be a breakthrough pass, improve the score of this action, so as to improve the number of breakthrough passes. After adding the classifier, the test is carried out. In the face of the 2021 version of some RoboCup teams, the test winning rates before and after the modification are as follows:

comparison of test results

4 Further work

In recent years, more and more progress has been made in intensive learning and has solved some problems such as GO, DOTA, StarCraft, etc. From simulation games, human-computer games to robot arm control, automation system, and even to recommendation engine and natural language dialogue system on Web system, the application of reinforcement learning is everywhere.

RoboCup is a sequence decision-making problem, and it is also Suitable to use reinforcement learning to improve. but the current problems in Reinforce learning include credit distribution, large scale and low sampling efficiency. In order to better promote RoboCup soccer and strengthen the learning community to carry out cooperation and exchange. based on RoboCup Simulation 2D platform SoccerMARL is proposed to develop a team coordination strategy intelligent decision system through deep reinforcement of learning and multi-agent cooperation. The project uses the deep neural network trained by actor critical method to evaluate the behavior selection in the robot tactical cooperation, and makes the analysis and decision of the action sequence. Finally, the feasibility and application value of multi-agent cooperation are explored through experiments.

References

1. Hidehisa Akiyama and Tomoharu Nakashima. Helios base: An open source package for the robocup soccer 2d simulation. In Sven Behnke, Manuela Veloso, Arnoud Visser, and Rong Xiong, editors, RoboCup 2013: Robot World Cup XVII, pages 528–535, Berlin, Heidelberg, 2014. Springer Berlin Heidelberg.
2. Fang Baofu. Robot soccer simulation [M]. Hefei University of Technology Press, 2011
3. Xu Xiaoxing. Formation and pass coordination of 2D simulation soccer robot system [D]. Nanjing: Nanjing University of Posts and telecommunications, 2016
4. Nagae N, Igarashi H, Ishihara S. Return Pass in RoboCup 2D Simulation League: State Prediction and Evaluation Function of the Chain Action[C]// Proceedings of the Fuzzy System Symposium. 2015. Iran:Shiraz University.2016.
5. Zhou Zhihua. Machine learning: = machine learning [M]. Tsinghua University Press, 2016
6. Long Defu. Mining and application of RoboCup Simulation 2D game log [D]. Guangdong: Guangdong University of technology, 2016
7. Li Xuejun. Local pass and tactical cooperation of soccer robot [D]. Nanjing: Nanjing University of Posts and telecommunications, 2016
8. Peng Jun, Wu Min, Cao Weihua, key technology of RoboCup robot soccer simulation competition [J]. Computer Engineering, 2004,30 (4): 49-51
9. Chen rongya. Research on player cooperation strategy in simulated robot soccer [D]. Hefei: University of science and technology of China, 2015
10. Circus, logistic regression model, blog Garden, [EB] / [OL], https://www.cnblogs.com/zm-pop-pk/p/11668294.html