View Original PDF

FCP_GPR_2015 Team Description Paper: Using Setplays and Self Adjusted positioning in simulated soccer teams.

João Alberto Fabro, Vinícius Brenner, Luis Paulo Reis, Nuno Lau

LASER Advanced Robotics and Embedded Systems Lab., Federal University of Technology – Paraná (UTFPR), Curitiba, Brazil; DSI/School of Engineering, University of Minho Guimarães, Portugal; LIACC Artificial Intelligence and Computer Science Lab., University of Porto, Portugal; IEETA, UA Inst. Eng. Elect and Telematics of Aveiro, University of Aveiro, Portugal

Abstract FCP_GPR_2015 is the result of joining forces between FCPortugal (from Portuguese universities of Aveiro, Minho and Porto) and GPR-2D (from Brazilian Federal University of Technology - UTFPR). FCPortugal have been participating in Robocup since 1999 (being the 2000 world champion in the 2D simulation category), and recently have focused its research objectives in multi-robot coordination methodologies in general, and predefined plays – Setplays – in specific, having recently released a complete set of tools to integrate setplays to soccer playing agents as free software (see SourceForge projects: fcportugalsetplays for the C++ Setplay Library, fcportugalSPlanner for a Graphical Interface to design Setplays, and fcportugalsetplaysagent2d for a complete example of 2D simulation agent). GPR2D won the Brazilian 2D competition in 2011, and have participated in Robocup since 2012, using machine learning techniques to improve the decision making of the robotic agents. Since 2014, these two teams joined their efforts into FCP_GPR, a team that seeks to unite the best features of each research group into one competitive team. This TDP presents the latest attempts to join Setplays and machine learning into an integrated approach.

Keywords: Simulated Robot Soccer, Coordination, Machine learning, Setplays.

1 - Introduction

FCP_GPR_2015 is the result of joining forces between FCPortugal [1] (from Portuguese universities of Aveiro, Minho and Porto) and GPR-2D [2] (from Brazilian Federal University of Technology – UTFPR). FCPortugal is a research initiative from the Portuguese research centers at the universities of Minho (DSI - Information Systems Department), Aveiro (IEETA – Institute of Electrical Engineering and Telematics), and Porto (LIACC - Artificial Intelligence and Computer Science Lab). This team of researchers participate in Robocup since 1999, in various categories (Rescue, 2D and 3D soccer simulation, Small Size, Standard Platform and Medium Size League). Among other research goals, these institutes are developing techniques and tools for machine learning, control, coordination and cooperation among agents, with application to various competitive and collaborative tasks, among then the RoboCup Soccer and Rescue leagues.

GPR-2D is the Robotics Research Group simulation soccer team from UTFPR (Federal University of Technology – Paraná – Brazil). Together with other research groups from other universities of the south of Brazil (Federal University of Santa Catarina and State University of West Paraná) this team have been participating in Robocup since 2012, after winning the Brazilian national competition in 2011. Among its research objectives are the use of adaptive intelligent approaches to provide simulated and real robots with capabilities to cope with complex scenarios.

Since 2013, these two research groups are working closely to each other, and in Robocup 2014 they joined efforts participating with the joint FCP_GPR_2D team. This 2D simulation team joined the ongoing effort of predefined cooperation among agents (Setplays), that is an ongoing a research field from FCPortugal [2] [3] [4] [5], with machine learning approaches developed by GPR [1]. Among the results of these efforts, are the publication, as free software [6], of a set of tools to graphically create and adjust predefined Setplays. This paper briefly presents Setplays definitions and the complete Setplay Framework [6], in section 2. In section 3, the combination of Setplays with learning techniques (such as those used by GPR) are presented, and section 4 present some simulation results against teams from Robocup 2014.

2. Setplays in Robocup Soccer Simulation 2D

Setplays, or set pieces, are predefined plans used by many team in various sports, such as football and soccer. The objective is to surprise the adversary team, executing a previously rehearsed set of movements, passes, dribbles, trying to gain some advantage. Usually Setplays start from situations such as faults close to the adversary goal area, or corner kicks, and the main objective is to provide a clear opportunity of scoring a goal. The FCPortugal research team research lead to the development of a high-level, XML based, specification language for soccer Setplays [2][3][4]. After that, a Graphical Strategy Planner – SPlanner – was developed, to graphically design and adjust all Setplay characteristics, providing an easy-to-use way to specify complex Setplays [7]. These technologies have been successfully applied to at least 3 different Robocup soccer leagues: 2D and 3D simulation, as well as Middle Size League [8]. And finally, a complete example team for 2D soccer simulation, based on Agent2D 3.1.1 [9], was developed, that can be used by anyone interested in provide its team with Setplay capabilities[6].

A Setplay defines all the parameters of a predefined sequence of actions that should be executed synchronously by a set of participating agents. Among the parameters are the Setplay name, region where it starts, number of players involved, and the starting situation (Kick-off, Throw-in, Free Kick, or Corner). After that, a series of "Steps" define what each participating player should do at each time so that the collaboration can occur. In Fig. 1, it is presented the graphical application developed to help designing and specifying complete Setplays, SPlanner [8], together with an example of steps (presented at the bottom of the figure), and in Fig. 2 the initial part of the XML-like specification file generated by the "Export Setplay" option.

Fig. 1 - Setplay definition tool - SPlanner - and example of Setplay steps (bottom).
Fig. 1 - Setplay definition tool - SPlanner - and example of Setplay steps (bottom).

3 – Self-Adjusted Setplay positioning

GPR2D was developed on top of Agent2D source code [9]. The main idea was to use Machine Learning techniques, such as the Q-Learning algorithm [1] to provide learning capabilities to the soccer playing agents. At first, only the agent in possession of the ball updated the learning matrices in order to select the best action, and only when inside the opponent goal area. The reinforcement approach was used in order to attempt different possible options of play inside the adversary goal area, and the reinforcement occurred each time any agent scored a goal [1]. FCP_GPR2014 [10], last year's team, and the first using Setplays from FCPortugal together with machine learning from GPR, focused on an approach in which the Setplays had options multiple choices of options, and the Q-Learning approach was used to select the "best" action from these choices. Although the initial promising results [11], the bottom line showed that the adaptation of the team was very sensitive to the adversary, and even more sensitive to the fine-tunning of the Setplay itself. Only a very carefully adjusted Setplay allowed the team to correctly execute the coordinated steps, allowing the machine learning algorithm to successfully find sets of choices that lead to better results. Another setback related to the lack of generalization capabilities, i.e., after successful learning of the better options of a certain Setplay against a certain adversary team, the performance against different teams was not as good, and the re-adjustment of the Setplay positions where too time consuming. Based on that findings, for this year's competition, a different approach has been proposed: the adjustment of the Setplay positions by means of an automated procedure. The following subsection details this approach.

(setplay :name CornerKick1 :id 1 :invertible true
:players (list (Player P6) (Player P7) (Player P8) ) 
:steps
  (seq
     (step  :id 0 :waitTime 25 :abortTime 59
          :participants
                     (list   (at (Player P6) (pt :x 52 :y 34))
                                (at (Player P7) (pt :x 40 :y 28))
                                (at (Player P8) (pt :x 31 :y 21)) )
          :condition (and (playm ck_our) (bpos :region (regNamed :name left)))
          :leadPlayer (Player P6)
          :transitions
              (list  (nextStep :id 1
                         :condition (canPassPl :from (Player P6) :to (Player P7) )
                                 :directives (list
                                            (do  :players(list (Player P6) )
                                                      :actions(bto:player(Player P7))  )
                                            (do  :players(list (Player P7) )
                                                      :actions(intercept )  )
                                           (do  :players(list (Player P8) )
                                                      :actions(pos :region (pt :x 41 :y 12)) )
                                )
               )
          ).......

Fig. 2 – Initial section of the exported "setplay.conf" XML-like file

3.1 – The Proposed Approach: auto-adjustable Setplay positions.

Since one of the main difficulties with Setplays is to correctly "fine-tune" the positioning, the proposed approach is based on a "learning" approach to adjust the positions of each player in each step of the Setplay. In the example presented in Figs. 1 and 2, the positions (X and Y) of each player in each step of the Setplay (seen in boldface in Fig. 2) are manually set with SPlanner. But during its execution, many considerations should be taken in account: Are those positions correct? The opponent team is positioned so that the passes are possible? In order to cope with these variables, a representation of each position in the Setplay was extracted, generating a vector of positions. This vector is then used as parameter for an optimization algorithm (Hill Climbing [12]), that executes as follows:

  • The Setplay is evaluated, using 10 simulated games, in which an averaged evaluation of the Setplay execution is calculated. The "execution quality" of the Setplay depends on the average of steps correctly executed. Suppose that during the 10 games, 25 times the "Corner Kick" Setplay was started. In 10 executions, only step 2 was reached (after that, the team lost the control of the ball, or the ball was intercepted by adversaries). In other 10 executions, step 3 was reached. And only in 5 of the executions, step 4 was reached. The "execution quality" is calculated in equation (1) for this example:

$$\text{ExecQual} = (102 + 103 + 5*4)/25 \rightarrow 2.8$$

  • After the initial evaluation, a "perturbation" is performed in the positions composing the vector related to the Setplay. In the experiments, a perturbation of +/- 1.5 points, applied to 10% of the positions, where performed. This means that only one position, for example (pt :x 41 :y -12) would be changed to (pt :x 42.5 :y -12), in a Setplay with 10 positioning values in it.
  • The procedure would be repeated, and another 10 games are executed. If the "execution quality" of the Setplay improves, the perturbation is maintained, otherwise the perturbation is removed, and another random perturbation is generated. Although time consuming, this procedure allowed the auto-adjustment of the positions of the Setplays automatically.

Usually this procedure were repeated during a training session of one night(8 hours), using agent2D 3.1.1 as opponent. After tweaking with the "synch_mode" simulation parameter, and using a computer equipped with an i7 processor with 4 cores, and 3.2 GHz of clock, it allowed the execution of each match against Agent2D in less than 4 minutes (instead of the regular 10 minutes), increasing the number of matches per night to around 120 matches (or 12 "epochs") per training session.

4 – Preliminary Results and Discussion

FCP_GPR intents to join the strength of both FCPortugal and GPR research teams. By using together the Setplay library and tools, with reinforcement learning procedures, the joined team has improved its performance over both FCPortugal and GPR in previous competitions. For this year's competition, the approach focuses in one of the drawbacks of the team. Although GPR reinforcement learning approach has good results when the any player receives the ball inside the adversary goal area, it was common against strong teams that the ball never reached a player inside the goal area, due to the very good defensive behavior of teams such WrightEagle [13] and Helios [9]. After several optimizations of Setplays for Corner Kicks and Throw-Ins in different sections of the field, the capability of the team of reaching the adversary area improved significantly. In simulations against WrightEagle, last year's champion, the team obtained 20% of wins (won 4 games in 20 simulations, scoring 10 and taking 42 goals). Against last year's runner up, Gliders, the team obtained 30% of wins (won 6 in 20 simulations, scoring 21 and taking 31 goals). In matches against Agent2D 3.1.1, the results where of more than 90% of winning. The auto-adjustment effort is still ongoing, and improvements are expected in the near future, but the integration of predefined Setplays with auto-adjustment has presented interesting results so far.

Acknowledgements

The first author would like to thank CAPES, process No. BEX 9292/13-6. We would also like to acknowledge the Robocup community for its support, specially teams Helios [9] and CMUnited [14] for their excellent contributions to the community and source releases that are the base for our teams.

References

  1. Lau, N.; Reis, L. P.; Mota, L.; Almeida, F. FC Portugal 2D Simulation: Team Description Paper, online, available at: http://staff.science.uva.nl/~arnoud/activities/robocup/RoboCup2013/ Symposium/TeamDescriptionPapers/SoccerSimulation/Soccer2D/, consulted on Jan/2014.
  2. Neri, J.R.F.; Zatelli, M.R.; Farias dos Santos, C.H.; Fabro, J.A.; A Proposal of QLearning to Control the Attack of a 2D Robot Soccer Simulation Team, 2012 Brazilian and Latin American Robotics Symposium (SBR-LARS), pp.174-178, 16-19 Oct. 2012.
  3. Mota, L.; Lau, N.; Reis, L.P.; Co-ordination in RoboCup's 2D simulation league: Setplays as flexible, multi- robot plans, 2010 IEEE Conf. on Robotics, Automation and Mechatronics, RAM 2010, pp. 362-367.
  4. Mota, L.; Reis, L.P.; An Elementary Communication Framework for Open Co-operative RoboCup Soccer Teams, in Sapaty P; Filipe J (Eds.) 4th Int. Conf. on Informatics in Control, Automation and Robotics - ICINCO 2007, pp. 97-101, Angers, France, May 9-12, 2007
  5. Mota, L.; Reis, L.P.; A Common Framework for Cooperative Robotics: an Open, Fault Tolerant Architecture for Multi-league RoboCup Teams, Int. Conf. Simulation Modeling and Progr. for Aut. Robots (SIMPAR), Springer, LNCS/LNAI series, pp. 171-182, Venice, Italy, Nov, 2008.
  6. Mota, L. ; Fabro, J. A. ; Reis, L. P. ; Lau, N. Collaborative Behavior in Soccer: The Setplay Free Software Framework. In: The 18th annual RoboCup International Symposium, 2014, Joao Pessoa-PB.
  7. Cravo, J.; Almeida, F.; Abreu, P.H.; Reis, L.P. ;Lau, N; Mota, L. Strategy planner: Graphical definition of soccer set-plays, Data & Knowledge Engineering 94, pp. 110-131, Nov. 2014
  8. Lau, N.; Lopes, L. S.; Corrente, G. and Filipe, N.; Roles, Positionings and Set Plays to Coordinate a MSL Robot Team, Proc. 14th Port. Conf. on Artificial Intelligence, EPIA'2009, Aveiro, LNAI 5816, Springer, pp 323-337, October 12-15, 2009.
  9. Akiyama, H. Helios RoboCup Simulation League Team, online, available at: http://rctools.sourceforge.jp/pukiwiki/, consulted on: Jan/2015.
  10. Fabro, J. A.; Oenning, B. O.; Brenner, V.; Reis, L. P. ; Lau, N. FCP_GPR_2014 Team: Joining Setplays from FCPortugal with Reinforcement Learning from GPR2D to Improve Decision-Making in Multi-Option Setplays, online, available at: http://fei.edu.br/rcs/2014/ TeamDescriptionPapers/SoccerSimulation/Soccer2D/, consulted on Jan/2015.
  11. Fabro, J.A. ; Reis, L. P. ; Lau, N. Using Reinforcement Learning techniques to select the best Action in Setplays with multiple possibilities in Robocup Soccer Simulation teams. In: LARS'2014 - 11th Latin American Robotics Symposium and SBR'2014 - 2nd Brazilian Symposium on Robotics, 2014, São Carlos-SP.
  12. Russell, S. J.; Norvig, P. Artificial Intelligence: A Modern Approach (2 ed.). Pearson Education, 2003.
  13. Zhang, H., Lu, G; Chen, R.; Li, X.; Chen, X. WrightEagle 2D Soccer Simulation Team Description 2014, online, available at: http://fei.edu.br/rcs/2014/TeamDescriptionPapers/ SoccerSimulation/Soccer2D/, consulted on Jan/2015.
  14. Stone, P.; Riley, P.; Veloso, M. CMUnited-99 source code, 1999, online, available at: http://www.cs.cmu.edu/~pstone/RoboCup/CMUnited99-sim.html, consulted on Feb/2000.