R-ZWEI KICKERS – Team Description Paper 2024

Adrian Müller, Wilhelm Simus

University of Applied Sciences Kaiserslautern

www.hs-kl.de · https://github.com/AK-Smart-Machines-HS-KL/R2K-SPL/wiki

Abstract The R-ZWEI KICKERS, founded in 2020, established within four years a stable team, contributing actively to the field and having a positive impact on education and research at UAS Kaiserslautern. The foundation of our current development is the B-Human 2021 code release, which allowed us to quickly catch up with the league and make meaningful contributions. Our work in 2023 focused on behavior improvements, communication compression, reinforcement learning for strategy optimization, as well as new team strategies and shooting techniques. We will participate at the next RoboCup German Open 2024 as well as in the SPL Challenge Shield in the 2024 RoboCup World Cup in Eindhoven. Additionally, we will participate in the "Shared Autonomy Challenge".

Keywords: Hybrid AI, Audio Communication, Reinforcement Learning, Teach-in.

1 Team Information

Team Name: R-ZWEI KICKERS

Team Lead: Prof. Adrian Müller (Scientific Lead)

Wilhelm Simus (Team Captain)

Contact: [email protected]

Website: https://github.com/AK-Smart-Machines-HS-KL/R2K-SPL/wiki

Country: Germany

Affiliations: UAS Kaiserslautern

2 Code Usage

2.1 Current state

The code we use to make the robots play soccer is largely based on B-Human's 2021 Code Release [1]. While the behavior control architecture used by our team is taken directly from the B-Human code release, the actual behaviors, implemented as cards, are custom-built by us.

Since then, no other team's code has been used in our code base.

2.2 Anticipated usage

Whistle Detection. During the most recent competitions we have discussed our issues with Whistle Detection with members of other teams. The whistle triangulation we are currently developing is directly inspired by these conversations and therefore our upcoming solutions might be influenced or based on code by other teams, e.g. the Directional Whistle Detection developed by Frankfurt's Bembelbots.

3 Own Contributions

In the following section we shall briefly describe this team's code contributions to the soccer software within the last three years while also highlighting future development plans.

3.1 Behavior

Behavior steered by Hybrid AI. While B-Human's rule-based card concept and their linear, top-down dispatcher provide a relevant structural backbone to the development of our robots' behaviors in the beginning, the behaviors themselves within this framework were found difficult to develop and maintain. To mitigate these difficulties, our team came up with an approach using the concept of hybrid AI, wherein some standard behaviors are executed as implemented into cards within the priority list, while more specific situation-based behaviors are "taught" to the robots by a human "coach" using robotic teach-in methods. This results in another set of cards ("teach-in cards") that does not have to be tediously implemented, and the execution of this set is steered by our own breadth-first dispatcher. We will discuss the details in the following sections.

Architecture for human-robot interaction. To allow the human "coach" to teach a robot, we had to come up with an architecture that allows for appropriate human-robot interaction. Our approach can be summarized as follows:

• 1. Capture the current game state in a "word model" (a discrete representation of players and ball position, score, time, etc.))
• 2. During simulation the "coach" intervenes in a specific game situation, teaching a dedicated behavior ("playback sequence") for one manually selected robot and storing the corresponding word model and sequence in what we call a "teach-in card".
• 3. Later, offline, the world model data is subjected to a data mining analysis, which identifies a list of "trigger points". They span the search space for our breadth-first dispatcher.
• 4. During gameplay, the teach-in system searches for the best fitting trigger points and notes the corresponding playback sequences. Each qualified teach-in card will potentially be executed; we use a weighted random selection method.
• 5. Hybrid Execution: rule-based and teach-in cards are mixed into a single gameplay stack.

Reinforcement Learning for Strategy Optimization. In 2023, we finished our reinforcement learning environment. It basically covers the five steps discussed above, but replacing the human guidance (step 2) and the data mining analysis (step 3) with an automated "trial and error" learning mechanism. The mechanism consists of B-Human's SimRobot, a RF learning AI, and a socket-based communication protocol. The protocol allows the learning component to defined game scenes, start and stop games, read out the ground truth and score the game state.

However, as expected, the search space is huge. Therefore, in 2024 we will use the learning process primarily for fine-tuning specific scenarios.

Expected Development towards Future Events. Our focus is on inter-robot communication optimization (content x frequency x prioritization), coordinated behavior (passing, assuming role based dynamic positions, …), and high-level behavior programming aids (hybrid AI).

3.2 Communication

Event Based Communication. The team communication "EBC" module automatically monitors state changes in relative ball position, behavior, assigned roles, game phase and state, and team behavior. Important messages, e.g. ball detections, or intents to pass, are broadcasted with higher priority, while others are deferred, resulting in a dynamically pulsed team communication.

In any state of communication, there is a minimum number of messages sent to keep the ball and obstacle models sufficiently synchronized. The base frequency therefore is adjusted dynamically according to the remaining budget, computing a linear burndown forecast, while remaining a budget reserve (30 messages) to cope with synchronization anomalies in the last few seconds of game:

$$send_interval = \frac{defaultFlexibleInterval * getTotalSecsRemaining()}{getOwnTeamInfoMessageBudget - 30}$$

The initial broadcast interval per bot – the defaultFlexibleInterval - is estimated as

$$defaultFlexibleInterval = \frac{1200 \text{ messages}}{2*10 \text{ min}*60 \text{ sec}*5 \text{ bots}} = 5 \text{ secs/bot}$$

Our pulsed burn-down approached worked well in all tournaments. The flexible interval fluctuates in an average of less than 10% – unless many robots get picked-up or penalized during the game.

Compressed Team Communications. To comply with the message budget regulations of the SPL, we have downsized the size of the broadcasted messages, by leaving out components we deemed strategically negligible or not relevant enough. This can be configured at compile time and is subject for further optimizations. Furthermore, we have devised a method to compress these messages using the Bitpacker package.

Expected Development towards Future Events. Besides optimizing the compression of team communication, we have recently started developing a protocol geared towards rudimentary "verbal" inter-robot communication using "beep" sounds encoded by their respective frequencies. Pre RoboCup 2023 we successfully testing a prototype that recognized one sound. We integrate the prototype towards executing the audio protocol, mapping meaningful messages to the beeps, and interfacing the messages with the card dispatcher. The idea is to be able to quickly shout and react to the most needed commands like "Leave the ball, I got it" or "Ball is right behind you" in real time, with no delay and zero bandwidth usage from our pulsed WLAN protocol.

3.3 Motion and Kicks

In the past years we did not yet implement our own functionality with respect to motion control, but took usage of the B-Human motion engine of 2021. We started changing that in 2023 with our own long shot skill on top the existing motion engine. Recently, we experiment with new features (esp. kick-variants, side-step) of our own.

Expected Development towards Future Events. In the realm of motion, in the near future we plan on utilizing simple arm and leg movements in order to implement some prototypes towards non-verbal inter-robot communication (pointing and gestures) as well as some gimmicks like an after-match victory dance (to please the audience). Last, but not least, we are working on improving our soccer gameplay by taking use of the freshly developed kicks in specific strategical behavior, utilizing them for an more elaborated game play.

3.4 Perception

Whistle Detection. While our code base has become somewhat robust in detecting when a referee whistle has been blown, we encountered incorrect detections during last year's competitions, due to matches being held in parallel on near-by fields. Children's shrill laughter and screams are another source of problems here, leading to false positives. We achieved a better quality by applying a higher sample rate and larger input size to the Fourier Transform, which allows for more fine-grained signal detection, reducing the possibility for false positives.

Considering the incorrect detections, we try to triangulate the origin (direction) of the sound. In addition to that, we plan to identify a ranking of whistles within audio detection range of a reasonable threshold, distinguish them by their signature and volume, and solely listen to the most like candidate for the rest of the game.

4 Past History

The main competition results of recent years can be viewed in Table 1.

So far, the R-ZWEI KICKERS have participated in the following events, starting with remote participation in technical challenges and steadily working towards the main competitions:

• German vRoHOW Workshop 2020
• 2021 German workshop on image annotation
• German Open Robotic Workshop RoDEO 2021, played 5 vs. 5 remotely, and conducted two local challenges on our own premises [6]
• German Open Replacement Event (GORE) 2021, several 5 vs. 5 remote games [7]
• RoboCup 2021 SPL, in two local challenges: Obstacle Avoidance and Passing Challenge [8]
• GORE 2022 in Hamburg where we entered the second round in the SPL [9]
• RoboCup 2022, remotely participating in the Video Challenge
• At GORE 2023 in Hamburg, we ranked 4th [10]
• At RoboCup SPL 2023, Challenge Shield, we ranked 2nd , and 5th in the Dynamic Ball Handling Challenge [11]

Results in the RoboCup main competition (5 vs. 5) in recent years. Games in bold are games we won. Rows in cursive show summarized information

5 Impact

The R-ZWEI KICKERS team is part of the Smart Machines working group [2] at UAS Kaiserslautern. We take an agile approach to development. Students from all faculties are invited to join [3]. In Summer term 2023, we hosted three Indian students abroad, from Presidency University, India.

Additionally, we frequently showcase our work for groups of pupils, freshmen, and visitors [4], thus promoting RoboCup and its events locally.

For higher semester students, we offer courses on AI and Robotics [5]. In Winter Term 2023/24, we welcomed several beginner students from a new study program – Digital Engineering - to our team.

All team members are encouraged to join SPL workshops, challenges, and tournaments.

Our industrial partner abat partially sponsors our participation in these events. The major impact of our joint work is the successful identification of technical questions (e.g. machine learning, vision, simulation, HCI, …) and possible solution strategies.

6 External challenges

Unfortunately, our progress this past year was held back by a cyber-attack on the university's infrastructure in June 2023. Shortly afterwards GORE 2023 the team was not able to use their resources at the university for several weeks, forcing us to prepare for the world championship by primarily making sure that our personal hardware was safe and not subject to said cyber-attack and secondly use the code we had downloaded in our different local development branches to rebuild our repository on GitHub, so were able to continue development toward the competition.

Since then, the university is operational again, but we have decided to continue future development on GitHub [3]. We transferred our code base (with some gaps in branch histories) and the Wiki - which we overhauled in the process - there.

To enable other teams to replicate our code or use it for strategical or development purposes, we have published a code release containing the state of our codebase shortly after RoboCup 2023 [12], such that other teams can use it as they see fit.

Acknowledgments. Special thanks go to Andy Hobelsberger, our team captain in 2023, who formed the team, made several important modifications to our code base (compressed communication, added several skills and cards, and – last, but not least – set up our successful infrastructure for the tournaments and actively hacked us over any in-competition-hurdles we encountered. "The fun is at the tournament"). Good luck with your new robot project back home in Chicago, Andy

The authors also like to thank Connor Lismore, David Kostka, Thomas Jäger and many more, unnamed students for their valuable contributions.

And with a "tip o´the hat" our thanks go to

• Philipp Stopp, our active sponsor and tactical soccer expert
• Asfiya Aazim Shirsangi (CS), Bangalore: Deployment, Whistle Detection
• Mohammed Nooruddin Asrar (CS), Bangalore: Tactics, Log Analysis
• Nishay Anand (CS): Deployment

References

1. B-Human Code Release 2021, https://github.com/bhuman/BHumanCodeRelease/tree/coderelease2021, last accessed 2021/02/15.
2. R-ZWEI KICKERS Team Page, https://smart-machines.hs-kl.de/projekte/hs-kl-robocup
3. Public Wiki and git repository: https://github.com/AK-Smart-Machines-HS-KL/R2K-SPL
4. R-ZWEI KICKER's public event calendar, https://smart-machines.hs-kl.de/veranstaltungen/, last accessed 2024/02/15.
5. Müller, Adrian "Robotics", Lecture and module description, 2022, https://campusboard.hskl.de/portalapps/sv/Modul_17412.pdf?modulid=17412&stgid=460
6. RoHOW 2021 web page, https://rohow.de/2021/, last accessed 2024/02/14.
7. GORE 2021 web page, https://www.robocup.de/en/2021/05/08/562/
8. Results of RoboCup 2021, https://spl.robocup.org/results-2021, last accessed 2024/02/14.
9. GORE 2022 web page, www.robocup.de/gore-2022/, last accessed 2024/02/14.
10. GORE 2023 results, https://gore-event.github.io/, last accessed 2024/02/14.
11. RoboCup 2023 results, https://spl.robocup.org/results-2023/, last accessed 2024/02/14.
12. R-ZWEI KICKERS' code release 2023, https://github.com/AK-Smart-Machines-HS-KL/R-ZWEI-Kickers-Code-Release-2023, last accessed 2024/02/14.