View Original PDF

BahiaRT 2019: Team Description Paper for RoboCup 3D Soccer Simulation League

Abstract This paper describes the most recent improvements for team BahiaRT in RoboCup 3D Soccer Simulation League since 2018. Our main focus was on setplays development. We describe both hand-coded setplay and some machine learning strategies used in our research. We also describe our work on enhancement of our path planning solution.

Introduction

Team BahiaRT has experienced a continuous evolution in RoboCup competitions in last years. In the world championship, BahiaRT ranked fifth in 2014, sixth in 2017 and fourth in 2015, 2016 and 2018. In the last four years, we were also Champions(in 2015) and runner-up(2016) of the Portuguese RoboCup Open competition. We won the last six editions of Brazilian Robotics Competition and the last five editions of Latin American Robotics Competition.

Aiming to maintain this evolution, we have decided to enhance some of our high-level AI strategies as well as increase some basic skills. We have worked enhancement of our path planning strategy. The main idea is to make the player who is dribbling the ball to choose a path towards the opponent goal that avoid opponent players as much as it is possible (see Section 2). We have continued or work with setplays. We developed and tested during last competitions a setplay for kickoff situations. It was hand-coded and reused most of our previous behaviors. We are now working on using some existing tools like SPlanner and its corresponding framework [4] to create a complete library of setplays. We have also started a project to use some machine learning strategies to use human knowledge to create and enhance setplays [11]. Section 3 describe our work about setplays.

Since 2013, team BahiaRT is based on a source base code named FCPBase 2013, as part of an international scientific cooperation partnership with team FC Portugal from Universities of Aveiro and Porto in Portugal.

Path planning

In the latest years we have managed to evolve significantly our teams pathplanning mechanics. The present high-level planning allow us to define a final target (usually the ball), and also set temporary targets using the active agent current position as a reference, along with any possible obstacle position that might be in its path to its final destination.

These temporary targets allows the agent to move dynamically, avoiding possible obstacles if necessary. A further development is still needed since there are a few problems yet to be solved in the teams navigation, like when the agent might get lost when near the limits of the football field, or when it might not choose the best route depending on its location or its target location. An example for that is when there is a large number of agents in his current path but for some reason he does not change it to avoid possible collisions. To solve this situations, a few solutions involving reinforced learning techniques [6], studies on potential fields [1] or even the utilization of other center of mass calculation models like the Linear Inverted Pendulum model [5] are in analysis. Our goal is to enhance our path planning engine up to RoboCup 2019 so we can test and validate its efficiency during matches in the world championship.

Fig. 1. Example of implemented path planning. The red line points to the final target, the blue one points to the temporary target and the green circles show the possible active agents. At that point there were two possible active agents, but eventually they would choose who was the best as the other would return to its strategic position.
Fig. 1. Example of implemented path planning. The red line points to the final target, the blue one points to the temporary target and the green circles show the possible active agents. At that point there were two possible active agents, but eventually they would choose who was the best as the other would return to its strategic position.
Fig. 2. Here we can see that the temporary target is set to position the agent behind the ball first in order for him to push it to the goal.
Fig. 2. Here we can see that the temporary target is set to position the agent behind the ball first in order for him to push it to the goal.

Setplays

Setplays are plans to create set pieces responsible for providing intelligent and cooperative behavior to the team[8]. This concept have been used in several RoboCup soccer leagues during last decade[9][7][3][2].

A setplay is a flexible and multi-step plan involving a variable number of robots [9]. A setplay has a name or numeric identification and can be shared by a set of players. During any setplay activation, the list of participants points to the subset of players involved in this specific instance. These player references can point to specific players or player roles. A set of abort conditions define rules to abort setplay at any time. The setplay roles were renamed to steps. Steps are the main building blocks of a setplay. Setplays can contain an arbitrary number of steps gathered in a list. A step can be seen as a state in the execution of a setplay. The agents participating in a setplay may execute all steps or just some of them. Each step has a set of conditions named transitions that define rules to move from one step to another. These rules are based on properties sensed by the agents from the environment or temporal conditions. There are two possible temporal conditions: wait time and abort time. Wait time is the amount of time an agent should wait before changing to the next step. Abort time is a deadline when the setplay is aborted with failure. Finish conditions that end a setplay with successful execution are also defined. Each step has a numeric identification (id). For convention step 0 is the initial step.

Each step defines the behaviors that each participant agent should execute during that specific step. These are active behaviors fired by specific critical condition rules defined for each step. A setplay can be modeled as a deterministic finite state automaton (DFA). To provide coordination between agents, a communication and synchronization policy was defined [9]. Each step will be led by the so-called lead agent. Generally, the lead agent is the one who makes most of the decisions in a step. In robotic soccer domain, the agent with ball possession can be the lead agent of a step in setplay which the team has ball possession. The lead agent broadcasts messages to all participants containing information such as setplay id, current step, step transitions, setplay abort. This is a simple way to keep all participants synchronized during setplay execution.

In the last two years we have decided to perform some research using setplays in our 3D soccer simulation team. We are using the graphical tool SPlanner[4] and its companion framework[9][7] to turn easier to create new setplays.

The Figure 3 represents a setplay drawn in the SPlanner interface, in step one, player 6 passes the ball to player 7, then player 7 passes the ball for the player eight, and finally the eight player kicks to the goal finishing the move. This is an example of a situation that often occurs in real games and is a good opportunity to score a goal.

Our first efforts were to integrate all these tools into our 3D Soccer Simulation agent[10]. As most of the previous work using these tools were done in 2D Soccer Simulation league or Medium Size league, we had to adapt some properties to work fine in our 3D Soccer Simulation agent. This work led us to build our first simple kickoff setplay[10]. Now we are working in an enhanced kickoff setplay and changing the SPlanner and framework to support other behaviors and requirements for development of better setplays. We are also starting the use of some machine learning strategies to enable agents to learn new setplays.

Fig. 3. Example of setplay in SPlanner.
Fig. 3. Example of setplay in SPlanner.

3.1 Kickoff setplay

Initially, we have developed an offensive kickoff setplay by hand with no support of SPlanner and its framework. We decided to do this way because many requirements we need to integrate SPlanner to our team were not ready. So we developed this setplay in parallel to SPlanner integration[10].

The setplay is fired when the play mode is an offensive Kickoff. It is divided into 4 steps (Figure 4) and its main goal is to take the ball as closer as possible to the opponent goal area while retaining the ball possession with our team.

In step 1, player 11 goes towards the ball to kick the ball back towards player number 6 and the player number 11 begins to move to its final position in front of the opponent goal area. In step 2, player number 6 waits for the ball to arrive to execute the pass for the players who advance to the entrance of the opponent goal area. Players advancing are 7, 8 and 11. While player 6 waits for the ball, players 9 and 10 perform an offensive marking behavior so that the opposing player can not anticipate the ball before it reaches the player number 6. The offensive marker consists in positioning in a middle point in the trajectory between the closest opponent player and the ball.

In step 3, player number 6 selects one of the 3 players which are near the opponent goal area and performs a long pass to the chosen player. In step 4, after one of the selected players receives the ball the setplay is finished with success and the team continues its usual offensive behavior.

The results obtained with this setplay were excellent because all times the setplay is finished with success, our team often manages to keep the ball in the opponent's defense field for much longer besides creating great chances of scoring goals. The setplay is not successful when one of the passes (short pass done by player 11 to player 6 or long pass done by player 6 to one of the advanced attackers) fail or when the offensive markers cannot avoid opponents to anticipate the ball before it reaches player number 6.

Fig. 4. Setplay for offensive kickoffs.
Fig. 4. Setplay for offensive kickoffs.

3.2 Adapting SPlanner and Framework

Some modifications are being made to the SPlanner and framework to meet the requirements of team BahiaRT. It was analyzed that, in order to perform the use of this tool[10], it was necessary to make some updates. The SPlanner lacks some important actions and behaviors existing in team BahiaRT. We have added these behaviors to make it feasible to use SPlanner to create new setplays.

The new actions are:

  • DirectWalk a faster and more practical gait than the defaut walk behavior;
  • ShortKick a short range and high precision kick;
  • OffensiveMarker a behavior to prevent an opponent take the ball possession from our team;
  • PasstoBestPlayer a new behavior that maps the original Pass Behavior of team BahiaRT

Another important modification was the inclusion of opponent players in the field. This way we can use some conditions such as distance to the nearest opponent to define step transitions or even abort or finish conditions. The Figure 5 demonstrates the addition of the opposing team in the SPlanner tool.

Fig. 5. Addition of the opposing team in blue in the top right side of the field.
Fig. 5. Addition of the opposing team in blue in the top right side of the field.

3.3 Learning Setplays from human experience

The use of auxiliary tools such as SPlanner and its framework turn the development of new setplays easier and fastest than programming everything by hand. Although, we continue to notice some situations during matches where players would perform another setplay that was not thought before by the development team. A soccer expert watching 3D Soccer Simulation league matches would also have a better idea for a setplay than those created by the development team.

We have started a research project to use human experts knowledge to build a dataset which can be used for agents to learn new setplays [11]. The main idea is to change Roboviz to create a demonstration mode. In this mode, the human expert can pause a match whenever he wants. At this point, he can position the exact situation when he thinks a new setplay should be used (by using reward or forward buttons). When the start situation is ok, he will click on the New Setplay Button and launch a SPlanner screen initialized with the current scene in Roboviz.

He will use SPlanner to define the setplay he wants to recommend and it will be saved as a record in the dataset. When lots of records defined by many human experts are available we can use this dataset to make the agents learn new useful setplays. The Figure 6 shows the main idea of this project.

This is a work in progress and we expect to use its preliminary results in RoboCup 2019.

Fig. 6. Proposal for Setplays Learning from human experience.
Fig. 6. Proposal for Setplays Learning from human experience.

Acknowledgements

This project is partly funded by CNPq/PIBIC, FAPESB/IC, UNEB/PICIN. We thank team FCPortugal for their cooperation and partnership.

References

  1. Chen, Y.b., Luo, G.c., Mei, Y.s., Yu, J.q., Su, X.l.: Uav path planning using artificial potential field method updated by optimal control theory. Intern. J. Syst. Sci. 47(6), 1407–1420 (Apr 2016), http://dx.doi.org/10.1080/00207721.2014. 929191
  2. Fabro, J., Reis, L., Lau, N.: Using Reinforcement Learning Techniques to Select the Best Action in Setplays with Multiple Possibilities in Robocup Soccer Simulation Teams. In: D.f., B.K.R.R.M.V.F. (ed.) Proceedings - 2Nd Sbr Brazilian Robotics Symposium, 11Th Lars Latin American Robotics Symposium and 6Th Robocontrol Workshop on Applied Robotics and Automation, Sbr Lars Robocontrol 2014 - Part of the Joint Conference on Robotics and Intelligent Systems, Jcris 2014. pp. 85–90 (2015), citations: crossref, scopus
  3. João Alberto Fabro, Vinícius Brenner, L.P.R., Lau, N.: FCP GPR 2015 Team Description Paper: Using Setplays and Self Adjusted positioning in simulated soccer teams. In: Proceedings of RoboCup 2015 (2015)
  4. João Guilherme Bettencourt Cravo, Luís Paulo Reis, F.A.: Uma Aplicação Gráfica de Definição Flexível de Jogadas Estudadas no RoboCup. Master's thesis, Faculdade de Engenharia da Universidade do Porto (2011)
  5. KAJITA, S.: The 3d linear inverted pendulum mode : A simple modeling for a biped walking pattern generation. Proc. IEEE Conf. Intelligent Robots and Systems, Hawaii, USA, 2001 pp. 239–246 (2001), https://ci.nii.ac.jp/naid/ 10025654464/en/
  6. Kim, D., Lee, J., Sentis, L.: Robust dynamic locomotion via reinforcement learning and novel whole body controller. CoRR abs/1708.02205 (2017), http://arxiv. org/abs/1708.02205
  7. Mota, L., Fabro, J.A., Reis, L.P., Lau, N.: Collaborative behavior in soccer: The setplay free software framework. In: Robot Soccer World Cup. pp. 709–716. Springer (2014)
  8. Mota, L., Reis, L.P.: Setplays: Achieving coordination by the appropriate use of arbitrary pre-defined flexible plans and inter-robot communication. In: Proceedings of the 1st international conference on Robot communication and coordination. p. 13. IEEE Press (2007)
  9. Mota, L., Reis, L.P., Lau, N.: Multi-robot coordination using setplays in the middlesize and simulation leagues. Mechatronics 21(2), 434–444 (2011)
  10. RAMOS, C.E.d.R.: Planejador Multiagentes para criação de jogadas ensaiadas em um time de futebol de robôs simulados. Bachelor thesis, Universidade do Estado do Bahia, Salvador, Bahia, Brazil (2017), in Portuguese.
  11. Simões, M.A.C., Nogueira, T.: Towards Setplays Learning in a Multiagent Robotic Soccer Team. In: 2018 Latin American Robotic Symposium, 2018 Brazilian Symposium on Robotics (SBR) and 2018 Workshop on Robotics in Education (WRE). pp. 277–282 (Nov 2018)