View Original PDF

RoboCanes

Saminda Abeyruwan, Victor Diaz, Armando Locay, Nabeel Sneij, Andreas Seekircher, Justin Stoecker, Ubbo Visser

Department of Computer Science, University of Miami, Coral Gables, FL, USA

Abstract Our team, RoboCanes, was formed in January 2010, shortly before the qualification deadline for RoboCup 2010. One of the team members was the former team leader of Virtual Werder 3D (VW3D) from the University of Bremen. His move to the University of Miami induced taking the original source code from VW3D and porting it to the new requirements of the soccer server and robot model. The intended research activities of RoboCanes follow those of VW3D in the area of behavior/situation recognition and prediction. The current activities are divided into short-term activities to be addressed before RoboCup 2010 and mid-term activities beyond this competition. Recent changes in the 3D simulation league towards humanoid simulation forces the teams to work on a number of new basic skills. The short-term goals address the development of low-level skills. The mid-term goal of the team is to apply plan recognition methods in order to bring valuable knowledge into the behavior decision process. The application of learning methods for learning low-level skills as well as higher-level behaviors is another research direction addressed by the team.

Introduction

Our team, RoboCanes, was formed in January 2010, shortly before the qualification deadline for RoboCup 2010. One of the team members was the former team leader of Virtual Werder 3D (VW3D) from the University of Bremen. His move to the University of Miami induced taking the original source code from VW3D and porting it to the new requirements of the soccer server and robot model.

The intended research activities of RoboCanes follow those of VW3D in the area of behavior/situation recognition and prediction. Our current activities can be divided into two parts: short-term activities to be addressed before RoboCup 2010 and the mid-term activities beyond this competition.

Recent changes in the 3D simulation league towards humanoid simulation forces the teams to work on a number of new basic skills. Thus, our short-term goals address the development of low-level skills as presented in section 2. The mid-term goal of our team is to apply plan recognition methods in order to bring valuable knowledge into the behavior decision process. These efforts are presented in section 3. The application of learning methods for learning low-level skills as well as higher-level behaviors is another research direction addressed by our team presented in section 4.

Humanoid Walking Engine and Special Actions

The development of the robot's basic skills in the RoboCanes agent is based on the experiences and results of the Bremen humanoid team B-Human [RBF+07] (a follow-up from the BreDoBrothers, which was a joint team from the Universität Bremen and the Universität Dortmund [RFH+06]). This is an important step towards merging research efforts of two separate RoboCup leagues. The 3D simulation league with the new server version can benefit from the experiences of the real robot humanoid league. Later on, a sufficiently realistic simulation can be used to ease certain aspects during the development of real robots by (pre-) learning some skills or testing different settings in the simulation that might be disadvantageous (and costly) for real robots.

In the first step, existing technologies of the B-Human team are integrated into the RoboCanes agent. It is tested if and how these technologies can be used in the simulation league's environment. The first skill that has been implemented is the walking engine (as presented in the RoboCanes binary for the RoboCup qualification); for more information about the walking engine see [NRL07,LR06,RFH+06]. In order to use the walking engine, the dimensions and physical properties of the simulated agent have to be provided. Furthermore, the agent's status of the different joints must be passed to the walking engine and the resulting effector command have to be mapped to the corresponding effectors in the simulation.

The B-Human team has developed a number of further so called "special actions" like:

  • getting up,
  • walking backwards,
  • walking left / right,
  • kicking the ball (with the left or right leg).

These special actions are also to be tested on the simulated robot and adapted. It is not expected that these special actions work out of the box. After some major parameter adaptions in order to create a first version of the intended behavior, fine tuning of the parameters has to be done in a second phase. We are planning to apply automated optimization methods like genetic algorithms [Mit98,PLM08,Gol89] or reinforcement learning [Wil92,SB98b] in order to identify good settings for the different actions. Further optimizations are planned as described in section 4.

The experiences gained from the integration, adaption, and optimization of the actions in the simulation should then flow back to the real robot team in the next step, which hopefully can be helpful to improve the performance of the real robots.

Behavior/Situation Recognition

A persistent research direction of our working group addresses the recognition of intentions and plans of agents. Such high-level functions cannot be used before a coordinated control of the agent is possible. Substantial advances have been made in past few years experimenting and developing various techniques such as logic-based approaches [WLV09,WV09], approaches based on probabilistic theories [Rac08], and artificial neural networks [Sta08]. The results have not been fully implemented in the current code yet. The current 3D server settings and performance (especially for a larger number of robots) lowers the probability of a fully functional behavior recognition and prediction method for a team of agents. We will follow this research approach nevertheless as a mid-term goal knowing that these issues will become important shortly.

Our approach to plan recognition is based on a qualitative description of dynamic scenes (cf. [WSV03,WVH05,DFL+04,MVH04]). The basic idea is to map the quantitative information perceived by the agent to qualitative facts that can be used for symbolic processing. Given a symbolic representation it is possible to define possible actions with their preconditions and consequences. In previous work real soccer tactical moves as, for instance, presented in Lucchesi [Luc01], have been formalized [Bog07]. As planning algorithms themselves are costly and thus hard to use in a demanding online scenario as robotic soccer, previously generated generic plans are provided to the agent who then can select the best plan w.r.t. some performance measure out of the set of plans that can be applied to a situation. As the pre-defined plans take into account multiagent settings it is possible to select a tactical move for a group of agents where different roles are assigned to various agents. In the 2D simulation league and the previous server of the 3D simulation league this approach has already been applied as behavior decision component in some test matches [WBE07,Bog07].

We propose a set of tools for spatio-temporal real-time analysis of dynamic scenes. It is designed to improve the grounding situation of autonomous agents in (simulated) physical domains. We introduced a knowledge processing pipeline ranging from relevance-driven compilation of a qualitative scene description to a knowledge-based detection of complex event and action sequences, conceived as a spatio-temporal pattern matching problem. A methodology for the formalization of motion patterns and their inner composition is defined and applied to capture human expertise about domain-specific motion situations. We present extensive experimental results from a challenging environment: 3D soccer simulation. It substantiates real-time applicability of our approach under tournament conditions, based on 5 Hz a) precise and b) noisy/incomplete perception. It is important to note that the approach is not limited to robot soccer. Instead, it can also be applied in other fields such as experimental biology and logistics processes [WLV09].

Our research is partly an application of the concepts developed in the parallel project "Automatic Recognition of Plans and Intentions of Other Mobile Robots in Competitive, Dynamic Environments" (research project in the German Research Councils priority program "Cooperating Teams of Mobile Robots in Dynamic Environments") to the new 3D soccer server. It is necessary to identify a set of relevant strategic moves that can be either applied by the own team (if the probability for a successful move is high) or recognized from observing the behavior of the opponent team. The German Research Council (DFG) supported our research line since 2001 (ended with move to US) and invited us to submit ideas for further long-term research ideas in that area. This clearly indicates the significance of our research efforts. Currently, several research proposals have been submitted or are in preparation (e.g. NSF, NIH, and internal UM proposals).

Reinforcement Learning

Reinforcement learning is a popular method in the context of agents and learning where a reward is given to an agent in order to evaluate its performance and thus, (hopefully) learning an optimal policy for action selection [Wil92,SB98b]. Reinforcement learning has been applied successfully in robotic soccer before by other teams (e.g., [MR02,RGH+06,KS04]). We have integrated a framework for reinforcement learning into our agent where different variants like Q-Learning and SARSA can be used (cf. [Wat89,WD92,SB98a]).

It is planned to apply reinforcement learning at two different levels: First of all, we want to investigate how certain skills can be optimized by reinforcement learning, e.g., in order to walk faster or to stand up in shorter time. It is focused to this kind of learning tasks until the RoboCup in Singapore.

The second level where learning should be applied is located in the behavior decision process. If it is known which strategic moves are possible the selection of the preferable move should be learned by reinforcement learning methods. The set of possible actions is determined by the applicable plans. The reward is given w.r.t. to the result of plan execution, e.g., if it failed or if it could be finished successfully. The desired result would be an automatically optimized high-level behavior based on a set of pre-defined plans. Different experiments have to show how the performance of the team can be improved in matches with identical or varying opponent teams.

Acknowledgments

We would also like to thank the former Virtual Werder 3D team, namely Tobias Warden, Carsten Rachuy, and Arne Stahlbock for the conception and implementation of the Virtual Werder code. We also would like to thank members of the B-Human team – namely, Thomas Röfer, Tim Laue, and Cord Niehaus – for providing their walking engine and the support integrating it into our agent.

References

  1. [Bog07] Tjorben Bogon. Effiziente abduktive Hypothesengenerierung zur Erkennung von taktischem und strategischem Verhalten im Bereich RoboCup. Master's thesis, Universitaet Bremen, 2007. Submitted.
  2. [DFL+04] F. Dylla, A. Ferrein, G. Lakemeyer, J. Murray, O. Obst, T. Röfer, F. Stolzenburg, U. Visser, and T. Wagner. Towards a League-Independent Qualitative Soccer Theory for RoboCup. In RoboCup 2004: Robot Soccer World Cup VIII. Springer, 2004.
  3. [Gol89] David E. Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning. Kluwer Academic Publishers, Boston, MA., 1989.
  4. [KS04] G. Kuhlmann and P. Stone. Progress in 3 vs. 2 keepaway. In RoboCup-2003: Robot Soccer World Cup VII, pages 694 – 702. Springer Verlag, Berlin, 2004.
  5. [LR06] T. Laue and T. Röfer. Getting upright: Migrating concepts and software from four-legged to humanoid soccer robots. In E. Menegatti E. Pagello, C. Zhou, editor, Proceedings of the Workshop on Humanoid Soccer Robots in conjunction with the 2006 IEEE International Conference on Humanoid Robots, 2006.
  6. [Luc01] M. Lucchesi. Coaching the 3-4-1-2 and 4-2-3-1. Reedswain Publishing, 2001.
  7. [Mit98] M. Mitchell. An introduction to genetic algorithms. The MIT press, 1998.
  8. [MR02] A. Merke and M. Riedmiller. Karlsruhe Brainstormers a reinforcement learning way to robotic soccer. In RoboCup 2001: Robot Soccer World Cup V, pages 435–440. Springer, Berlin, 2002.
  9. [MVH04] Andrea Miene, Ubbo Visser, and Otthein Herzog. Recognition and prediction of motion situations based on a qualitative motion description. In D. Polani, B. Browning, A. Bonarini, and K. Yoshida, editors, RoboCup 2003: Robot Soccer World Cup VII, LNCS 3020, pages 77–88. Springer, 2004.
  10. [NRL07] C. Niehaus, T. Röfer, and T. Laue. Gait optimization on a humanoid robot using particle swarm optimization. In C. Zhou, E. Pagello, E. Menegatti, and S. Behnke, editors, IEEE-RAS International Conference on Humanoid Robots, 2007.
  11. [PLM08] R. Poli, WB Langdon, and N.F. McPhee. A field guide to genetic programming. Lulu Enterprises Uk Ltd, 2008.
  12. [Rac08] Carsten Rachuy. Erstellen eines probabilistischen Modells zur Klassifikation und Prädiktion von Spielsituationen in der RoboCup 3d Simulationsliga. Master's thesis, Universität Bremen, (forthcoming) 2008.
  13. [RBF+07] Thomas Röfer, Christoph Budelmann, Martin Fritsche, Tim Laue, Judith Müller, Cord Niehaus, and Florian Penquitt. B-Human team description for RoboCup 2007, 2007.
  14. [RFH+06] Thomas Röfer, Martin Fritsche, Matthias Hebbel, Thomas Kindler, Tim Laue, Cord Niehaus, Walter Nistico, and Philippe Schober. BreDoBrothers team description for RoboCup 2006, 2006.
  15. [RGH+06] M. Riedmiller, T. Gabel, R. Hafner, S. Lange, and M. Lauer. Die brainstormers: Entwurfsprinzipien lernfähiger autonomer roboter. Informatik-Spektrum, 29(3):175–190, June 2006.
  16. [SB98a] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press, Cambridge, MA, 1998. Online version available at: http://www.cs.ualberta.ca/~sutton/book/ebook/the-book.html.
  17. [SB98b] R.S. Sutton and A.G. Barto. Introduction to reinforcement learning. 1998.
  18. [Sta08] Arne Stahlbock. Sitationsbewertungsfunktionen zur Unterstutzung der Aktionsauswahl in der 3D Simulationsliga des RoboCup. Master's thesis, Universität Bremen, (forthcoming) 2008.
  19. [Wat89] Christopher J. C. H. Watkins. Learning from delayed rewards. PhD thesis, University of Cambridge, England, 1989.
  20. [WBE07] Thomas Wagner, Tjorben Bogon, and Carsten Elfers. Incremental generation of abductive explanations for tactical behavior. Submitted to the RoboCup International Symposium 2007, 2007.
  21. [WD92] Christopher J. C. H. Watkins and Peter Dayan. Technical note: Q-learning. Machine Learning, 8(3-4):279–292, May 1992.
  22. [Wil92] R.J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229–256, 1992.
  23. [WLV09] Tobias Warden, Andreas Lattner, and Ubbo Visser. Real-time spatiotemporal analysis of dynamic scenes in 3d soccer simulation. In Luca Iocchi, Hitoshi Matsubara, Alfredo Weitzenfeld, and Changjiu Zhou, editors, RoboCup 2008: Robot Soccer World Cup XII, volume 5399 of Lecture Notes of Artificial Intelligence, pages 366–378. Springer, Heidelberg, 2009.
  24. [WSV03] T. Wagner, C. Schlieder, and U. Visser. An extended panorama: Efficient qualitative spatial knowledge representation for highly dynamic environments. In Proceedings of the IJCAI-03 Workshop on Issues in Designing Physical Agents for Dynamic Real-Time Environments: World modelling, planning, learning, and communicating, pages 109–116, 2003.
  25. [WV09] Tobias Warden and Ubbo Visser. Spatio-temporal analysis of dynamic scenes: An approach for real time domains. Knowledge and Information Systems, 2009. under revision.
  26. [WVH05] T. Wagner, U. Visser, and O. Herzog. Egocentric qualitative knowledge representation for physical robots. Journal for Robotics and Autonomous Systems, 2005.