View Original PDF

FC Portugal 3D Simulation Team: Team Description Paper 2022

Miguel Abreu, Mohammadreza Kasaei, Luís Paulo Reis, Nuno Lau

LIACC/LASI/FEUP, Artificial Intelligence and Computer Science Lab, Faculty of Engineering of the University of Porto, University of Porto, Portugal; School of Informatics, University of Edinburgh, UK; IEETA/LASI/DETI University of Aveiro, 3810-193 Aveiro, Portugal

Abstract The FC Portugal 3D team presented in this year's competition was rebuilt from the ground up in Python since the last RoboCup. No previous code was used or adapted, with the exception of the 6D pose estimation algorithm, and the get-up behaviors, which were re-optimized. In previous years, FC Portugal has contributed to research concerning low-level skills and high-level soccer coordination methodologies, both in the 2D and 3D soccer simulation leagues. The research-oriented development of our team has been pushing it to be one of the most competitive over the years with more than 30 international awards. This paper describes the team's new code, including an overview of our agent architecture and its main modules. All the new skills largely outperform previous ones: the omnidirectional walk is stabler and can reach 0.70 to 0.90 m/s, the long kick can reach on average 17 to 19 m, and the dribble is able to keep close control of the ball while moving between 1.25 and 1.41 m/s. We are also trying to increase student contact by providing reinforcement learning assignments to be completed using our new Python framework.

1 Introduction

Historically, FC Portugal has contributed to the simulation league (2D and 3D) in numerous ways, including competitive methodologies and server improvements. In previous years, one of the main research directions of our team was high-level decision and multi-agent cooperation mechanisms. For the RoboCup 3D soccer simulation competition that was based on spheres (from 2004 to 2006), the decisive factor (like in the 2D competition) was the high-level reasoning capabilities of the players, and not their low-level skills. Thus, we worked mainly on high-level coordination methodologies. Since 2007, humanoid agents have been introduced in the 3D simulation league, but the number of agents was kept small until 2011. During this period, research in coordination was not very important in the 3D league. Developing efficient low-level skills, contrarily to what should be the research focus of the simulation league, was the main challenge. However, in 2011, the number of agents increased to 9, and in 2012 to 11, making coordination, once again, one of the most important problems to create an efficient team.

Our research on high-level soccer coordination methodologies and team playing was mainly focused on the adaptation of previously developed methodologies from our 2D soccer teams [19, 25–27] to the 3D humanoid environment and on creating new coordination methodologies. Since the first participation in the 2D league, in RoboCup 2000, our team has introduced several concepts and algorithms covering a broad spectrum of the soccer simulation research challenges. Coordination techniques [6, 10, 18, 20, 22] such as tactics, formations, dynamic positioning and role exchange, situation-based strategic positioning, intelligent perception, optimization-based low-level skills [4, 5, 7, 11, 13, 14, 16, 17, 21, 23, 24, 28–35], sim-to-real transfer [12], optimization algorithms [1–3] and frameworks [8], visual debugging and coaching.

After RoboCup 2021, the team was rebuilt from the ground up in Python, without using or adapting previous code, with the exception of the 6D pose estimation algorithm, and the get-up behaviors, which were re-optimized. The new code is compatible with most data science libraries and machine learning repositories, allowing for fast development of new behaviors and tactics. Due to the improvement of hardware resources in recent years, developing a team in Python is no longer a major concern in terms of computational efficiency. However, some computationally demanding modules were written in C++ to ensure the agent does not lose control cycles.

2 Agent Architecture

The agent receives internal data every 20 ms, and visual data every 60 ms. The latter is a noisy partial view of the world state, which is fed to a 6D pose estimator to extract the robot's localization and orientation in a three-dimensional space [9]. Forward kinematics is then used to estimate the pose of every body part for a given robot type. This self-awareness ability in conjunction with team communication allows each agent to have a reliable representation of the world state. The following paragraphs will describe the main components of the agent.

2.1 Mid-level skills

The team has four major skills:

– Kick: A short kick is mainly used for passes, with a range between 3 and 9 m. The long kick is generally used for shooting, and has an average distance of 17 to 19 m, depending on the robot type;

  • Walk: The walk skill is omnidirectional and is able to sustain an average linear speed between 0.70 and 0.90 m/s, depending on robot type and walking direction;
  • Dribble: The dribble skill is able to push the ball forward while retaining close control. While dribbling forward, the robot reaches an average maximum speed of 1.25 to 1.41 m/s, depending on robot type;
  • Get Up: The robot uses this skill to get up after falling to the ground. There are three variations of the skill per robot type depending on the falling direction (front, back, sides).

All skills were optimized through reinforcement learning. The Kick, Walk, and Dribble skill are parameterized by neural networks and the Get Up by behavior slots. Each slot in the Get Up behavior corresponds to a pose the robot must execute for a specific period of time. Several slots can create complex movements that can easily be adjusted manually or optimized through machine learning.

2.2 Low-level control

The low-level control consists of an inverse kinematics module, used by the Walk and Dribble, and a 1-step predictive controller, which is indirectly used by the same skills, and directly used by the Kick and Get Up. Since the server sends the observations with a 1-step delay, the current observations can only be estimated by combining the previous ones with the last sent actions.

2.3 Team communication

The team uses three consecutive messages to send all the desired information. Accordingly, there are three distinct groups of variables, each containing different groups of players from our team and the opponent's team. One of the messages also contains the position of the ball. In general, each player is represented by 3 variables: its absolute x and y coordinates, as seen by the sender, and a state indicating whether the player is down or not. Each message is only sent if a given player is currently able to see the entire group of players. Incomplete groups are also allowed in special nonconsecutive communication rounds. This type of information is important to allow each agent to have a more complete view of the world state, since several agent modules rely on this knowledge, e.g. formation, role manager, path planning, etc.

2.4 Role Manager

The role manager is responsible for dynamically assigning roles to each player. The formation contains the desired position of each role, not the current position of each teammate. Some roles are only applicable in specific cases. The active player (AP) goes to the ball; AP2 and AP3 are also responsible for chasing the ball when the team is defending (i.e. the opponent has possession); the goalkeeper (GK) can become an active player and push the ball forward like any other player, although it will return to the goal once it loses possession; the central back (CB) is always between the ball and the goal; man-marking roles (MM) mark the closest 4 players of the opposing team, with different priorities according to their distance to our goal; the left and right support (LS, RS) aid the AP when the team is attacking, in case it loses possession; and the left and right front (LF, RF) roles create space to receive passes from the AP.

Roles have different priorities depending on the current team mentality (attacking or defending). In both cases, the highest priority role is the AP followed by the GK. While attacking, the front roles are the next most important to allow game progression, followed by the remaining roles. However, when defending, the support roles are converted to active players, until the team regains possession, and the front roles become the least important.

2.5 Path Planning

The path search algorithm is based on A* [15] and divides the soccer field into a grid (32 m × 22 m, 10 cm grid size). Obstacles include players, goalposts, goal nets, and sometimes the ball, depending on the agent's current objective. While dribbling, the outer field lines are also considered obstacles to keep the ball from going out of play. Each obstacle is defined by a position, a hard inaccessible radius, and a soft radius, which increases the cost of the path as it gets closer to the obstacle. The size of this radius depends on the desired aggressiveness, since each agent can decide to be careful, typically when moving to a strategic position, or aggressive, when disputing the ball. Teammates are also avoided by an agent, using larger margins, if their role is more important than said agent.

2.6 Main Routine

The main routine is divided into 5 major steps: decide an intention, such as moving to a point, dribbling, kicking, get up, etc; then choosing a skill, since some intentions can require different skills for positioning and execution; computing a target for the current skill, including a desired position or orientation, according to the skill requirements; execute the skill through a neural network or slot behavior; and finally broadcast information as explained in Section 2.3.

When the robot is standing, deciding an intention requires an analysis of promising passes, shooting directions and dribbling paths. The algorithm scores the potential outcomes to estimate what would be the best alternative for game progression. If none of these options is viable, the robot will push the ball by walking towards it until it has enough space to take other actions.

3 Conclusion

FC Portugal has developed numerous skills and methodologies concerning the NAO humanoid robot and the simulation league in general. Currently, the team has a very robust code base that was developed from scratch after RoboCup 2021. The development speed motivated by the new Python framework allowed us to develop stable and efficient skills in a short period of time. An integrated learning approach guarantees that skills such as the omnidirectional walk, dribble and kick can attain high performance but also smooth transitions, without falling or requiring intermediate steps. Despite the considerable gain in performance and competitiveness, in comparison with previous years, there are still many improvement opportunities. Future research directions include high-level multi-agent coordination strategies, opponent modeling, goalkeeper skills, omnidirectional kicks, optimization algorithms, and more.

References

  1. Abdolmaleki, A., Lau, N., Reis, L.P., Neumann, G.: Non-parametric contextual stochastic search. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 2643–2648. IEEE (2016)
  2. Abdolmaleki, A., Lioutikov, R., Peters, J.R., Lau, N., Paulo Reis, L., Neumann, G.: Model-based relative entropy stochastic search. Advances in Neural Information Processing Systems 28 (2015)
  3. Abdolmaleki, A., Simoes, D., Lau, N., Reis, L.P., Neumann, G.: Contextual relative entropy policy search with covariance matrix adaptation. In: 2016 International Conference on Autonomous Robot Systems and Competitions (ICARSC). pp. 94–99. IEEE (2016)
  4. Abdolmaleki, A., Sim˜oes, D., Lau, N., Reis, L.P., Neumann, G.: Learning a humanoid kick with controlled distance. In: Robot World Cup. pp. 45–57. Springer (2016)
  5. Abreu, M., Lau, N., Sousa, A., Reis, L.P.: Learning low level skills from scratch for humanoid robot soccer using deep reinforcement learning. In: 2019 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC). pp. 256–263. IEEE (2019)
  6. Abreu, M., Reis, L.P., Cardoso, H.L.: Learning high-level robotic soccer strategies from scratch through reinforcement learning. In: 2019 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC). pp. 128–134. IEEE (2019)
  7. Abreu, M., Reis, L.P., Lau, N.: Learning to run faster in a humanoid robot soccer environment through reinforcement learning. In: Robot World Cup. pp. 3–15. Springer (2019)
  8. Abreu, M., Rossetti, R.J., Reis, L.P.: XSS: a soccer server extension for automated learning of high-level robotic soccer strategies. In: 2019 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC). pp. 135–142. IEEE (2019)
  9. Abreu, M., Silva, T., Teixeira, H., Reis, L.P., Lau, N.: 6D localization and kicking for humanoid robotic soccer. Journal of Intelligent & Robotic Systems 102(2), 1–25 (2021)
  10. Certo, J., Lau, N., Reis, L.P.: A generic multi-robot coordination strategic layer. In: Proceedings of the 1st international conference on Robot communication and coordination. pp. 1–7 (2007)
  11. Cruz, L., Reis, L.P., Lau, N., Sousa, A.: Optimization approach for the development of humanoid robots' behaviors. In: Ibero-American Conference on Artificial Intelligence. pp. 491–500. Springer (2012)
  12. Domingues, E., Lau, N., Pimentel, B., Shafii, N., Reis, L.P., Neves, A.J.: Humanoid behaviors: from simulation to a real robot. In: Portuguese Conference on Artificial Intelligence. pp. 352–364. Springer (2011)
  13. Ferreira, R., Reis, L.P., Moreira, A.P., Lau, N.: Development of an omnidirectional kick for a NAO humanoid robot. In: Ibero-American Conference on Artificial Intelligence. pp. 571–580. Springer (2012)
  14. Ferreira, R., Shafii, N., Lau, N., Reis, L.P., Abdolmaleki, A.: Diagonal walk reference generator based on fourier approximation of zmp trajectory. In: 2013 13th International Conference on Autonomous Robot Systems. pp. 1–6. IEEE (2013)
  15. Hart, P., Nilsson, N., Raphael, B.: A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics 4(2), 100–107 (1968). https://doi.org/10.1109/tssc.1968.300136, https://doi.org/10.1109/tssc.1968.300136
  16. Kasaei, M., Abreu, M., Lau, N., Pereira, A., Reis, L.P.: Robust biped locomotion using deep reinforcement learning on top of an analytical control approach. Robotics and Autonomous Systems 146, 103900 (2021)
  17. Kasaei, S.M., Sim˜oes, D., Lau, N., Pereira, A.: A hybrid zmp-cpg based walk engine for biped robots. In: Iberian Robotics conference. pp. 743–755. Springer (2017)
  18. Lau, N., Lopes, L.S., Corrente, G., Filipe, N.: Multi-robot team coordination through roles, positionings and coordinated procedures. In: 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems. pp. 5841–5848. IEEE (2009)
  19. Lau, N., Reis, L.: FC Portugal High-Level Coordination Methodologies in Soccer Robotics, pp. 167–192. INTECH Open Access Publisher (12 2007). https://doi.org/10.5772/5130
  20. Lau, N., Reis, L.P., Certo, J.: Understanding dynamic agent's reasoning. In: Portuguese Conference on Artificial Intelligence. pp. 542–551. Springer (2007)
  21. Marques, H., Lau, N., Reis, L.P.: FC Portugal 3D simulation team: Architecture, low-level skills and team behaviour optimized for the new robocup 3d simulator. In: Proc. Scientific Meeting of the Portuguese Robotics Open. pp. 31–37. Citeseer (2004)
  22. Mota, L., Reis, L.P.: Setplays: Achieving coordination by the appropriate use of arbitrary pre-defined flexible plans and inter-robot communication. In: Proceedings of the 1st international conference on Robot communication and coordination. pp. 1–7 (2007)
  23. Picado, H., Gestal, M., Lau, N., Reis, L.P., Tom´e, A.M.: Automatic generation of biped walk behavior using genetic algorithms. In: International Work-Conference on Artificial Neural Networks. pp. 805–812. Springer (2009)
  24. Rei, L.: Optimizing Simulated Humanoid Robot Skills. Master's thesis, University of Porto, Porto (2010)
  25. Reis, L.P., Lau, N.: FC Portugal team description: Robocup 2000 simulation league champion. In: RoboCup 2000: Robot Soccer World Cup IV. pp. 29–40. Springer (2000)
  26. Reis, L.P., Lau, N.: COACH UNILANG a standard language for coaching a (robo) soccer team. In: RoboCup 2001: Robot Soccer World Cup V. pp. 183–192. Springer (2001)
  27. Reis, L.P., Lau, N., Oliveira, E.C.: Situation based strategic positioning for coordinating a team of homogeneous agents. In: Workshop on Balancing Reactivity and Social Deliberation in Multi-Agent Systems. pp. 175–197. Springer (2000)
  28. Shafii, N., Abdolmaleki, A., Ferreira, R., Lau, N., Reis, L.P.: Omnidirectional walking and active balance for soccer humanoid robot. In: Portuguese Conference on Artificial Intelligence. pp. 283–294. Springer (2013)
  29. Shafii, N., Lau, N., Reis, L.P.: Generalized learning to create an energy efficient zmp-based walking. In: Robot Soccer World Cup. pp. 583–595. Springer (2014)
  30. Shafii, N., Lau, N., Reis, L.P.: Generalized optimization and reinforcement learning for creating an energy efficient zmp walker. Proceedings of the 2014 Dynamic Walking, ETH Zurich (2014)
  31. Shafii, N., Lau, N., Reis, L.P.: Learning a fast walk based on zmp control and hip height movement. In: 2014 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC). pp. 181–186. IEEE (2014)
  32. Shafii, N., Lau, N., Reis, L.P.: Learning to walk fast: Optimized hip height movement for simulated and real humanoid robots. Journal of Intelligent & Robotic Systems 80(3), 555–571 (2015)
  33. Shafii, N., Reis, L.P., Lau, N.: Biped walking using coronal and sagittal movements based on truncated fourier series. In: Robot Soccer World Cup. pp. 324–335. Springer (2010)
  34. Sim˜oes, D., Amaro, P., Silva, T., Lau, N., Reis, L.P.: Learning low-level behaviors and high-level strategies in humanoid soccer. In: Iberian Robotics conference. pp. 537–548. Springer (2019)
  35. Teixeira, H., Silva, T., Abreu, M., Reis, L.P.: Humanoid robot kick in motion ability for playing robotic soccer. In: 2020 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC). pp. 34–39. IEEE (2020)