View Original PDF

RoboCanes Research focus

Saminda Abeyruwan, Andreas Seekircher, Justin Stoecker, Ubbo Visser

Department of Computer Science University of Miami Coral Gables, FL, USA

Abstract Our team, RoboCanes, was formed in January 2010, shortly before the qualification deadline for RoboCup 2010. One of the team members was the former team leader of Virtual Werder 3D (VW3D) from the University of Bremen. His move to the University of Miami induced taking the original source code from VW3D and porting it to the new requirements of the soccer server and robot model. Our research activities of RoboCanes follow those of VW3D in the area of behavior/situation recognition, prediction and control. Our current activities can be divided into two parts: short-term activities to be addressed before RoboCup 2011 and the mid-term activities beyond this competition.

Introduction

Our team, RoboCanes, was formed in January 2010, shortly before the qualification deadline for RoboCup 2010. One of the team members was the former team leader of Virtual Werder 3D (VW3D) from the University of Bremen. His move to the University of Miami induced taking the original source code from VW3D and porting it to the new requirements of the soccer server and robot model.

Our research activities of RoboCanes follow those of VW3D in the area of behavior/situation recognition, prediction and control. Our current activities can be divided into two parts: short-term activities to be addressed before RoboCup 2011 and the mid-term activities beyond this competition.

Recent changes in the 3D simulation league towards humanoid simulation forces the teams to work on a number of new basic skills. Thus, our short-term goals address the development of low-level skills as presented in section 2. The mid-term goal of our team is to apply plan recognition methods in order to bring valuable knowledge into the behavior decision process. These efforts are presented in section 3. The application of learning methods for learning low-level skills as well as higher-level behaviors is another research direction addressed by our team presented in section 4.

Humanoid Walking Engine and Special Actions

The development of the robot's basic skills in the RoboCanes agent is based on the experiences and results of the Bremen humanoid team B-Human [RBF+07] (a follow-up from the BreDoBrothers, which was a joint team from the Universität Bremen and the Universität Dortmund [RFH+06]). This is an important step towards merging research efforts of two separate RoboCup leagues. The 3D soccer simulation league with the new server version can benefit from the experiences of the real robot humanoid league. Later on, a sufficiently realistic simulation can be used to ease certain aspects during the development of real robots by (pre-) learning some skills or testing different settings in the simulation that might be disadvantageous (and costly) for real robots. In the first step, existing technologies of the B-Human team are integrated into the RoboCanes agent. It is tested if and how these technologies can be used in the simulation league's environment. The first skill that has been implemented is the walking engine (as presented in the RoboCanes binary for the RoboCup qualification); for more information about the walking engine see [NRL07,LR06,RFH+06]. In order to use the walking engine, the dimensions and physical properties of the simulated agent have to be provided. Furthermore, the agent's status of the different joints must be passed to the walking engine and the resulting effector command have to be mapped to the corresponding effectors in the simulation.

The B-Human team has developed a number of further so called "special actions" like:

  • getting up,
  • walking backwards,
  • walking left / right,
  • kicking the ball (with the left or right leg).

These special actions are also to be tested on the simulated robot and adapted. It is not expected that these special actions work out of the box. After some major parameter adaptions in order to create a first version of the intended behavior, fine tuning of the parameters has to be done in a second phase. We are planning to apply automated optimization methods like genetic algorithms [Mit98,PLM08,Gol89] or reinforcement learning [Wil92,SB98] in order to identify good settings for the different actions. Further optimizations are planned as described in section 4.

The experiences gained from the integration, adaption, and optimization of the actions in the simulation should then flow back to the real robot team in the next step, which hopefully can be helpful to improve the performance of the real robots.

Behavior/Situation Recognition

A persistent research direction of our working group addresses the recognition of intentions and plans of agents. Such high-level functions cannot be used before a coordinated control of the agent is possible. Substantial advances have been made in past few years experimenting and developing various techniques such as logicbased approaches [WLV09,WV11], approaches based on probabilistic theories [Rac08], and artificial neural networks [Sta08]. The results have been partly implemented in the current code. For a big portion of last year, the 3D server settings and performance (especially for a larger number of robots) lowered the probability of a fully functional behavior recognition and prediction method for a team of agents. The latest implementation of SimSpark however has changed this situation significantly so that we can follow this research approach as a short-term goal.

Our approach to plan recognition is based on a qualitative description of dynamic scenes (cf. [WSV03,WVH05,DFL+04,MVH04]). The basic idea is to map the quantitative information perceived by the agent to qualitative facts that can be used for symbolic processing. Given a symbolic representation it is possible to define possible actions with their preconditions and consequences. In previous work real soccer tactical moves as, for instance, presented in Lucchesi [Luc01], have been formalized [Bog07]. As planning algorithms themselves are costly and thus hard to use in a demanding online scenario as robotic soccer, previously generated generic plans are provided to the agent who then can select the best plan w.r.t. some performance measure out of the set of plans that can be applied to a situation. As the pre-defined plans take into account multiagent settings it is possible to select a tactical move for a group of agents where different roles are assigned to various agents. In the 2D simulation league and the previous server of the 3D simulation league this approach has already been applied as behavior decision component in some test matches [WBE07,Bog07].

We propose a set of tools for spatio-temporal real-time analysis of dynamic scenes. It is designed to improve the grounding situation of autonomous agents in (simulated) physical domains. We introduced a knowledge processing pipeline ranging from relevance-driven compilation of a qualitative scene description to a knowledge-based detection of complex event and action sequences, conceived as a spatio-temporal pattern matching problem. A methodology for the formalization of motion patterns and their inner composition is defined and applied to capture human expertise about domain-specific motion situations. We present extensive experimental results from a challenging environment: 3D soccer simulation. It substantiates real-time applicability of our approach under tournament conditions, based on 5 Hz a) precise and b) noisy/incomplete perception. It is important to note that the approach is not limited to robot soccer. Instead, it can also be applied in other fields such as experimental biology and logistics processes [WLV09].

Our research is partly an application of the concepts developed in the parallel project "Automatic Recognition of Plans and Intentions of Other Mobile Robots in Competitive, Dynamic Environments" (research project in the German Research Councils priority program "Cooperating Teams of Mobile Robots in Dynamic Environments") to the new 3D soccer server. It is necessary to identify a set of relevant strategic moves that can be either applied by the own team (if the probability for a successful move is high) or recognized from observing the behavior of the opponent team. The German Research Council (DFG) supported our research line since 2001 (ended with move to US) and invited us to submit ideas for further long-term research ideas in that area. This clearly indicates the significance of our research efforts. Currently, several research proposals have been submitted or are in preparation (e.g. NSF, NIH, and internal UM proposals).

Prediction and Control through Reinforcement Learning

Reinforcement learning is a popular method in the context of agents and learning where a reward is given to an agent in order to evaluate its performance and thus, (hopefully) learning an optimal policy for action selection [Wil92,SB98]. Reinforcement learning has been applied successfully in robotic soccer before by other teams (e.g., [MR02,RGH+06,KS04]). We have integrated a framework for reinforcement learning into our agent where different variants like Q-Learning and SARSA have been used (cf. [Wat89,WD92,SB98]). We have submitted our current as work to the RoboCup Symposium 2011 [ASVed].

It is planned to apply reinforcement learning at two different levels: First of all, we want to investigate how certain skills can be optimized by reinforcement learning, e.g., in order to walk faster or to stand up in shorter time. It is focused to this kind of learning tasks until the RoboCup in Singapore.

The second level where learning should be applied is located in the behavior decision process. If it is known which strategic moves are possible the selection of the preferable move should be learned by reinforcement learning methods. The set of possible actions is determined by the applicable plans. The reward is given w.r.t. to the result of plan execution, e.g., if it failed or if it could be finished successfully. The desired result would be an automatically optimized high-level behavior based on a set of pre-defined plans. Different experiments have to show how the performance of the team can be improved in matches with identical or varying opponent teams.

The recent learning tasks that have been carried out in the RL framework is based on linear function approximation, specially the penalty goal keep behavior. The reinforcement learning framework is extended with GQ(λ) and Greedy-GQ algorithms [MSBS10,BBSE10]. These algorithms have been proven to converge with linear function approximators and it is shown superior results in prediction and control problems.

Monitor and Debugging Tool

Justin Stoecker from our team RoboCanes has invented a new 3D soccer server monitor (RoboViz) that runs platform independent. RoboViz is a software program designed to assess and develop agent behaviors in a multi-agent system, the RoboCup 3D simulated soccer league. It is an interactive monitor that renders agent and world state information in a three-dimensional scene. In addition, RoboViz provides programmable drawing and debug functionality to agents that can communicate over a network.

The tool facilitates the real-time visualization of agents running concurrently on the SimSpark simulator, and provides higher-level analysis and visualization of agent behaviors not currently possible with existing tools (figure 1).

Features include visualization and debugging (e.g. real-time debugging; direct communication with agents; selecting shapes to be rendered), interactivity and control (e.g. reposition of objects; switching game-play modes), enhanced graphics (e.g. stereoscopic 3D graphics on systems with support for quad-buffered OpenGL; effects such as soft shadows and bloom post-processing provide a visually enticing experience), easy use (e.g. simple controls, automatic connection to the server, platform independency), and other features (e.g. various scene perspectives, logfile viewing, playback with different speeds). A detailed description of RoboViz has been submitted as a paper for the RoboCup Symposium [SVed].

RoboViz interface with debugging information and 2D bird view
RoboViz interface with debugging information and 2D bird view

SimSpark and ODE improvements

Sander van Dijk (Team Boldhearts) and our team RoboCanes are currently (Feb-Apr. 20011) working on a new SimSpark and ODE version. This work is supported by a RoboCup Federation Grant and is focussed on the following goals:

  1. Improve stability: fix bugs and increase robustness of simulator.
  2. Enable starting multiple instances on a single machine or over a network: make it possible to easily run multiple simulations in parallel. The result will be tested at the German Open 2011.
  3. Enhance run-time control: give the possibility to alter any simulation detail at run-time, alleviating need to constantly restart the system.
  4. Develop graphical utility tools: facilitate setting up a batch of experiments.

Sander has announced some of the new developments already in the mailing list.

Acknowledgments

We would also like to thank the former Virtual Werder 3D team, namely Tobias Warden, Carsten Rachuy, and Arne Stahlbock for the conception and implementation of the Virtual Werder code. We also would like to thank members of the B-Human team – namely, Thomas Röfer, Tim Laue, and Cord Niehaus – for providing their walking engine and the support integrating it into our agent.

References

  1. [ASVed] Saminda Abeyruwan, Andreas Seekircher, and Ubbo Visser. Reinforcement learning of soccer skills for humanoid robots. In Thomas Röfer, Norbert Michael Mayer, Jesus Savage, and Uluç Saranli, editors, RoboCup 2011: Robot Soccer World Cup XV, 2011, submitted.
  2. [BBSE10] Lucian Busoniu, Robert Babuska, Bart De Schutter, and Damien Ernst. Reinforcement Learning and Dynamic Programming Using Function Approximators. CRC Press, Inc., Boca Raton, FL, USA, 1st edition, 2010.
  3. [Bog07] Tjorben Bogon. Effiziente abduktive Hypothesengenerierung zur Erkennung von taktischem und strategischem Verhalten im Bereich RoboCup. Master's thesis, Universitaet Bremen, 2007. Submitted.
  4. [DFL+04] F. Dylla, A. Ferrein, G. Lakemeyer, J. Murray, O. Obst, T. Röfer, F. Stolzenburg, U. Visser, and T. Wagner. Towards a League-Independent Qualitative Soccer Theory for RoboCup. In RoboCup 2004: Robot Soccer World Cup VIII. Springer, 2004.
  5. [Gol89] David E. Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning. Kluwer Academic Publishers, Boston, MA., 1989.
  6. [KS04] G. Kuhlmann and P. Stone. Progress in 3 vs. 2 keepaway. In RoboCup-2003: Robot Soccer World Cup VII, pages 694 – 702. Springer Verlag, Berlin, 2004.
  7. [LR06] T. Laue and T. Röfer. Getting upright: Migrating concepts and software from four-legged to humanoid soccer robots. In E. Menegatti E. Pagello, C. Zhou, editor, Proceedings of the Workshop on Humanoid Soccer Robots in conjunction with the 2006 IEEE International Conference on Humanoid Robots, 2006.
  8. [Luc01] M. Lucchesi. Coaching the 3-4-1-2 and 4-2-3-1. Reedswain Publishing, 2001.
  9. [Mit98] M. Mitchell. An introduction to genetic algorithms. The MIT press, 1998.
  10. [MR02] A. Merke and M. Riedmiller. Karlsruhe Brainstormers a reinforcement learning way to robotic soccer. In RoboCup 2001: Robot Soccer World Cup V, pages 435–440. Springer, Berlin, 2002.
  11. [MSBS10] Hamid Reza Maei, Csaba Szepesvári, Shalabh Bhatnagar, and Richard S. Sutton. Toward off-policy learning control with function approximation. In ICML, pages 719–726, 2010.
  12. [MVH04] Andrea Miene, Ubbo Visser, and Otthein Herzog. Recognition and prediction of motion situations based on a qualitative motion description. In D. Polani, B. Browning, A. Bonarini, and K. Yoshida, editors, RoboCup 2003: Robot Soccer World Cup VII, LNCS 3020, pages 77–88. Springer, 2004.
  13. [NRL07] C. Niehaus, T. Röfer, and T. Laue. Gait optimization on a humanoid robot using particle swarm optimization. In C. Zhou, E. Pagello, E. Menegatti, and S. Behnke, editors, IEEE-RAS International Conference on Humanoid Robots, 2007.
  14. [PLM08] R. Poli, WB Langdon, and N.F. McPhee. A field guide to genetic programming. Lulu Enterprises Uk Ltd, 2008.
  15. [Rac08] Carsten Rachuy. Erstellen eines probabilistischen Modells zur Klassifikation und Prädiktion von Spielsituationen in der RoboCup 3d Simulationsliga. Master's thesis, Universität Bremen, (forthcoming) 2008.
  16. [RBF+07] Thomas Röfer, Christoph Budelmann, Martin Fritsche, Tim Laue, Judith Müller, Cord Niehaus, and Florian Penquitt. B-Human team description for RoboCup 2007, 2007.
  17. [RFH+06] Thomas Röfer, Martin Fritsche, Matthias Hebbel, Thomas Kindler, Tim Laue, Cord Niehaus, Walter Nistico, and Philippe Schober. BreDoBrothers team description for RoboCup 2006, 2006.
  18. [RGH+06] M. Riedmiller, T. Gabel, R. Hafner, S. Lange, and M. Lauer. Die brainstormers: Entwurfsprinzipien lernfähiger autonomer roboter. Informatik-Spektrum, 29(3):175–190, June 2006.
  19. [SB98] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 1998.
  20. [Sta08] Arne Stahlbock. Sitationsbewertungsfunktionen zur Unterstützung der Aktionsauswahl in der 3D Simulationsliga des RoboCup. Master's thesis, Universität Bremen, (forthcoming) 2008.
  21. [SVed] Justin Stoecker and Ubbo Visser. RoboViz: Programmable Visualization for Simulated Soccer. In Thomas Röfer, Norbert Michael Mayer, Jesus Savage, and Uluç Saranli, editors, RoboCup 2011: Robot Soccer World Cup XV, 2011, submitted.
  22. [Wat89] Christopher J. C. H. Watkins. Learning from delayed rewards. PhD thesis, University of Cambridge, England, 1989.
  23. [WBE07] Thomas Wagner, Tjorben Bogon, and Carsten Elfers. Incremental generation of abductive explanations for tactical behavior. Submitted to the RoboCup International Symposium 2007, 2007.
  24. [WD92] Christopher J. C. H. Watkins and Peter Dayan. Technical note: Q-learning. Machine Learning, 8(3-4):279–292, May 1992.
  25. [Wil92] R.J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229–256, 1992.
  26. [WLV09] Tobias Warden, Andreas Lattner, and Ubbo Visser. Real-time spatiotemporal analysis of dynamic scenes in 3d soccer simulation. In Luca Iocchi, Hitoshi Matsubara, Alfredo Weitzenfeld, and Changjiu Zhou, editors, RoboCup 2008: Robot Soccer World Cup XII, volume 5399 of Lecture Notes of Artificial Intelligence, pages 366–378. Springer, Heidelberg, 2009.
  27. [WSV03] T. Wagner, C. Schlieder, and U. Visser. An extended panorama: Efficient qualitative spatial knowledge representation for highly dynamic environments. In Proceedings of the IJCAI-03 Workshop on Issues in Designing Physical Agents for Dynamic Real-Time Environments: World modelling, planning, learning, and communicating, pages 109–116, 2003.
  28. [WV11] Tobias Warden and Ubbo Visser. Spatio-temporal analysis of dynamic scenes: An approach for real time domains. Knowledge and Information Systems, 2011. Accepted for publication.
  29. [WVH05] T. Wagner, U. Visser, and O. Herzog. Egocentric qualitative knowledge representation for physical robots. Journal for Robotics and Autonomous Systems, 2005.