FRA-UNIted — Team Description 2022

Thomas Gabel, Yalcin Eren, Fabian Sommer, Alexander Vieth, Eicke Godehardt

Faculty of Computer Science and Engineering Frankfurt University of Applied Sciences 60318 Frankfurt am Main, Germany

Abstract The main focus of FRA-UNIted's effort in the RoboCup soccer simulation 2D domain is to develop and to apply machine learning techniques in complex domains. In particular, we are interested in applying reinforcement learning methods, where the training signal is only given in terms of success or failure. In this paper, we review some of our recent efforts taken during the past year, also analyzing the impact that the ongoing Covid pandemic has been having on our team's activities.

1 Introduction

The soccer simulation 2D team FRA-UNIted is a continuation of the former Brainstormers project which has ceased to be active in 2010. The ancestor Brainstormers project was established in 1998 by Martin Riedmiller, starting off with a 2D team which had been led by the first author of this team description paper since 2005. Our efforts in the RoboCup domain have been accompanied by the achievement of several successes such as multiple world champion and world vice champion titles as well as victories at numerous local tournaments. Our team was re-established in 2015 at the first author's new affiliation, Frankfurt University of Applied Sciences, reflecting this relocation with the team's new name FRA-UNIted.

As a continuation of our efforts in the ancestor project, the underlying and encouraging research goal of FRA-UNIted is to exploit artificial intelligence and machine learning techniques wherever possible. Particularly, the successful employment of reinforcement learning (RL, [10]) methods for various elements of FRA-UNIted's decision making modules — and their integration into the competition team — has been and is our main focus. Moreover, the extended use of the FRA-UNIted framework in the context of university teaching has moved into our special focus. So, we aim at employing the 2D soccer simulation domain as a fundament for teaching agent-based programming, foundations of multi-agents systems as well as applied machine learning algorithms.

In this team description paper, we refrain from presenting approaches and ideas we already explained in team description papers of the previous years [3]. Instead, we focus on recent changes and extensions to the team as well as on reporting partial results of work currently in progress. We start this team description paper, however, with a short general overview of the FRA-UNIted framework. Note that, to this end, there is some overlap with our older team description papers including those written in the context of our ancestor project (Brainstormers 2D, 2005–2010) which is why the interested reader is also referred to those publications, e.g. to [6, 9].

1.1 Design Principles

FRA-UNIted relies on the following basic principles:

• There are two main modules: the world module and decision making
• Input to the decision module is the approximate, complete world state as provided by the soccer simulation environment.
• The soccer environment is modeled as a Markovian Decision Process (MDP).
• Decision making is organized in complex and less complex behaviors where the more complex ones can easily utilize the less complex ones.
• A large part of the behaviors is learned by reinforcement learning methods.
• Modern AI methods are applied wherever possible and useful (e.g. particle filters are used for improved self localization).

1.2 The FRA-UNIted Agent

The decision-making process of the FRA-UNIted agent is inspired by behaviorbased robot architectures. A set of more or less complex behaviors realize the agents' decision making as sketched in Figure 1. To a certain degree this architecture can be characterized as hierarchical, differing from more complex behaviors, such as "no ball behavior", to very basic, skill-like ones, e.g. "pass behavior". Nevertheless, there is no strict hierarchical sub-divisioning. Consequently, it is also possible for a low-level behavior to call a more abstract one. For instance, the behavior responsible for intercepting the ball may, under certain circumstances, decide that it is better to not intercept the ball, but to focus on more defensive tasks and, in doing so, call the "defensive behavior" and delegating responsibility for action choice to it.

2 Team-Internal Reflection in Regard to the Covid-19 Pandemic

Covid has been a predominant topic in society during the past two years. In this section, we elaborate on the influence that the ongoing pandemic has had on our robotic soccer simulation team.

2.1 Team Activity and Statistics

The Corona pandemic has had an enormous impact on our team's efforts in further developing our playing capabilities as well as on the implementation and integration of learning approaches into our competition team. Unfortunately, we have to recognize that we had to observe a severely declining interest (by university students) in participating in our team's activities as well as a declined activity among active team members. There are a number of reasons as well as a number of observable consequences that all more or less relate to changed circumstances due to the ongoing Corona situation.

This situation is best reflected by Figure 2 which shows the activity in our team's Git repository during recent years. Here, we have split the data into a "Pre-Covid Period" (before Covid hit Germany in March 2020) and a "Covid Period" (since March 2020). Apparently, the (coding-related) activity has dropped by more than 80% ever since (note that 2018 represents 100% in that statistic). We are fully aware that quantity and quality are two different things when it comes to implementing successful soccer simulation playing strategies. High activity in terms of many lines of code added or removed may not necessarily be a guarantuee for improved playing performance. However, a severe decrease in activity like the one shown here, is very likely to be corelated with less innovations introduced into the code base.

2.2 Teaching Experiment: Redevelop the Entire Play Without Ball

Given the difficult situation described in the previous section and knowing well that the effort for entering the soccer simulation domain is high, we performed an experiment in which we made students of a Bachelor level project course (part of the study B.Sc. programme on Computer Science), who had no prior contact to robotic soccer, redevelop an entire strategic (top-level) behavior, viz the "no ball behavior" depicted in Figure 1.

The students were provided an intensive initial training on the basics of soccer simulation 2D and on the FRA-UNIted code base (7 weeks) and were, subsequently, allowed to finish their task within up to 9 more weeks. Furthermore, they were provided an attenuated version of FRA-UNIted, called FU Base, where the mentioned no ball behavior was replaced by an 18-year-old predecessor version, but where the remaining parts remained intact (world model, coach, goalie as well as other behaviors on both, the strategic level and on the lower, skill-like levels). Although just one behavior has been removed¹, this weakened version is of significantly reduced playing strength compared to our team's full version (average result: 0.6:5.8) and even weaker than Agent2D [1] (average result: 2.1:4.6).

Two student teams (A and B) were formed, consisting of 5 and 6 students, respectively. The performance of both teams improved in the course of the 9 weeks of time provided (cf. Figure 3 to see how time was allocated across weeks by the teams), but failed to even reach the strength of FU_Base as can be seen from the right part of Figure 3.

3 Diversified RoboCup Training Protocols on a Distributed Architecture

Training Protocols With the development and implementation of the continuous integration environment, our RoboCup Team is constantly tested against the previous world champion or against other teams every night [5]. This process is producing statistics which are used to asses the performance of our team as well as the impact of changes made to our teams code. To further build on this approach, we are currently in the process of developing a training protocol system, which allows for sophisticated control over the training procedure. A training protocol is a server sided set of instructions which allows our team to play against multiple different opponents at the same time, as well as allowing for early cutoff thresholds which perform an action if, during training, a defined data point reaches a stable, user defined value. Furthermore these protocols are meant to be extendable with additional conditions or actions for the training process, such as automatically finding beneficiary values for team behavior parameters based on the games played under the respective protocol. We investigate a change to the already existing continuos integration procedure by letting the test PCs pull the current training protocol from a web server each night, which contains further instructions for the HLM² config file, team binaries and match data.

4 Aggressive Defense Behavior Utilizing Low-Level Opponent Action Predictions

We started to reimplement our one-on-one behavior, which was presented in 2008 (aka NeuroHassle [7]). Due to several adjustments of opposing teams and changes of the Soccer Server the original behavior became more and more inefficient over the years. For example, omni-directional movements of an agent are not included in the old behavior. The new behavior should be able to efficiently conquer the ball from a dribbling opponent moving towards our goal. This task is also realized with methods of reinforcement learning [10]. In addition, the results of our approach on opponent low-level action prediction [4] using a Case-Based Reasoning (CBR, [8]) approach are utilized to predict the opponent agent's behavior. So, the state of the game in the next time step can be more accurately predicted by our agent which enhances the capabilities of the RL algorithm to learn an even more competitive behavior.

First, we modeled the problem as a Markov Decision Process (MDP). The state of our agent is described as a nine-dimensional vector. Therefore, the center of the coordinate system was moved to the opponent's player and rotated by its orientation. The state is described by the

• distance d between our player and the opponent ball leading player,
• velocity ($v^x$ and $v^y$ component) of our player,
• absolute value $v_{opp}$ of the opponent's velocity,
• position ($b^x$ and $b^y$ component) of the ball,
• our player's body angle $\alpha$ relative to the opponent's position,
• opponent player's body angle $\beta$ relative to his direction towards our goal,
• value of the strategic angle $\gamma = \angle GOM$ with G as position of our goal, O as position of the opponent, and M as the position of our player.

Due the high dimensionality of the problem space we use a neural network to approximate our value function. The agent is allowed to use dash(power, α) and turn(α) commands with power ∈ [0, 10, . . . , 100] and α ∈ [−180, −165, . . . , 180]. The agent is rewarded when the ball comes into its kickable area or when their probability of executing a successful tackling exceeds 0.75. Additionally, the agent obtains a slight punishment for each time step it needs to reach the ball. This way we enforce learning a time-optimal behavior. The possible starting states for the training are defined in a similar manner as in [7]. In each episode, both players are starting with a random body angle and velocity. The opponent player is set in a circle around our player and has the ball in its kick area.

First Results At this stage, the algorithm is tested against the dribbling behavior currently used by our own team, which dribbles with the ball towards a given target point and which was learned using a reinforcement learning approach as well (Section 2.2 in [2]). When evaluating our preliminary results, we observed that the new one-on-one behavior shows an improvement in ball conquering of approximately 7% compared to the old behavior. Of course, training and learning against other top teams from recent RoboCup tournaments is one of the next steps to be taken. Final and stable results for learning this aggressive defense behavior in conjunction with an exploited reliable case-based prediction of opponent low-level actions are expected in the second quarter of this year and will hopefully be deployed during the upcoming RoboCup 2022 World Championships tournament in Bangkok (Thailand).

5 Conclusion

In this team description paper we have outlined the characteristics of the FRA-UNIted team participating in RoboCup's 2D Soccer Simulation League. We have stressed that this team is a continuation of the former Brainstormers project, pursuing similar and extended goals in research and development as well as for teaching purposes. Specifically, we have put emphasis on our most recent research activities and practical implementation of our results.

References

1. Akiyama, H.: Agent2D Base Code. https://osdn.net/projects/rctools/releases/ (2012)
2. Gabel, T., Breuer, S., Roser, C., Berneburg, R., Godehardt, E.: FRA-UNIted Team Description 2017 (2017), Supplementary material to RoboCup 2017: Robot Soccer World Cup XXI
3. Gabel, T., Breuer, S., Sommer, F., Godehardt, E.: FRA-UNIted Team Description 2019. In: RoboCup 2019: Robot Soccer World Cup XXIII, LNCS (CD Supplement). Springer, Sydney, Australia (2019)
4. Gabel, T., Godehardt, E.: I Know What You're Doing: A Case Study on Case-Based Opponent Modeling and Low-Level Action Prediction. In: Workshop on Case-Based Agents at the International Conference on Case-Based Reasoning (ICCBR-CBA 2015). pp. 13–22. Frankfurt (2015)
5. Gabel, T., Kloeppner, P., Eren, Y., Sommer, F., Breuer, S., Godehardt, E.: FRA-UNIted - Team Description 2021. In: RoboCup 2021: Robot Soccer World Cup XXV, LNCS (CD Supplement). Springer (2021)
6. Gabel, T., Riedmiller, M.: Brainstormers 2D Team Description 2009. In: L. Iocchi, H. Matsubara, A. Weitzenfeld, C. Zhou, editors, RoboCup 2009: Robot Soccer World Cup XII, LNCS (CD Supplement). Springer, Graz, Austria (2009)
7. Gabel, T., Riedmiller, M., Trost, F.: A Case Study on Improving Defense Behavior in Soccer Simulation 2D: The NeuroHassle Approach. In: L. Iocchi, H. Matsubara, A. Weitzenfeld, C. Zhou, editors, RoboCup 2008: Robot Soccer World Cup XII, LNCS. pp. 61–72. Springer, Suzhou, China (2008)
8. Richter, M.: Case-Based Reasoning. A Textbook. Springer, Heidelberg, Germany (2013)
9. Riedmiller, M., Gabel, T.: Brainstormers 2D Team Description 2007. In: U. Visser, F. Ribeiro, T. Ohashi and F. Dellaert, editors, RoboCup 2007: Robot Soccer World Cup XI, LNCS (CD Supplement). Springer, Atlanta, USA (2007)
10. Sutton, R.S., Barto, A.G.: Reinforcement Learning. An Introduction. MIT Press/A Bradford Book, Cambridge, USA (1998)