View Original PDF

UT Austin Villa 3D Simulation Soccer Team 2014

Patrick MacAlpine, Peter Stone

Department of Computer Science The University of Texas at Austin Austin, TX 78712-0233

Abstract This paper describes the research focus and ideas incorporated in the UT Austin Villa 3D simulation soccer team entering the RoboCup competitions in 2014.

Introduction

In this paper, we describe the agent our team UT Austin Villa is currently developing for participation at the 2014 RoboCup 3D Simulation Soccer competition. The main challenge presented by the 3D simulation league is the low-level control of a humanoid robot with more than 20 degrees of freedom. The simulated environment is a 3-dimensional world that models realistic physical forces such as friction and gravity, in which teams of humanoid robots compete with each other. Thus, the 3D simulation competition paves the way for progress towards the guiding goal espoused by the RoboCup community, of pitting a team of 11 humanoid robots against a team of 11 human soccer players. Programming humanoid agents in simulation, rather than in reality, brings with it several advantages, such as making simplifying assumptions about the world, low installation and operating costs, and the ability to automate experimental procedures. All these factors contribute to the uniqueness of the 3D simulation league.

The approach adopted by our team UT Austin Villa to decompose agent behavior is bottom-up in nature, comprising lower layers of joint control and inverse kinematics, on top of which skills such as walking, kicking and turning are developed. These in turn are tied together at the high level of strategic behavior. Details of this architecture are presented in this paper, which is organized as follows. Section 2 provides a brief overview of the 3D humanoid simulator. In Section 3, we describe the design of the UT Austin Villa agent, and elaborate on its skills in Section 4. In Section 5, we draw conclusions and present directions for future work.

Brief Overview of 3D Simulation Soccer

2007 was the first year of the 3D simulation competition in which the simulated robot was a humanoid. The humanoid used in the 2007 RoboCup competitions in Atlanta, U.S.A., was the Soccerbot, which was derived from the Fujitsu HOAP-2 robot model. Owing to problems with the stability of the simulation, the Soccerbot was replaced by the Aldebaran Nao robot at the 2008 RoboCup competitions in Suzhou, China. The robot has 22 degrees of freedom: six in each leg, four in each arm, and two in the neck and head. Figure 1 shows a visualization of the Nao robot and the soccer field during a game. The agent described in the following sections of this paper is developed for the Nao robot.

Each component of the robot's body is modeled as a rigid body with a mass that is connected to other components through joints. Torques may be applied to the motors controlling the joints. A physics simulator (Open Dynamics Engine) computes the transition dynamics of the system taking into consideration the applied torques, forces of friction and gravity, collisions, etc. Sensation is available to the robot through a camera mounted in its head, which provides information about the positions of objects on the field every third cycle. This information has a small amount of noise added to it and is also restricted to a 120° view cone. The visual information, however, does not provide a complete description of state, as details such as joint orientations of other players and the spin on the ball are not conveyed. Apart from the visual sensor, the agent also gets information from touch sensors at the feet and accelerometer and gyro rate sensors. The simulation progresses in discrete time intervals with period 0.02 seconds. At each simulation step, the agent receives sensory information and is expected to return a 22-dimensional vector specifying torque values for the joint motors.

Since 2007 was the year the humanoid was introduced to the 3D simulation league, the major thrust in agent development thus far has been on developing robotic skills such as walking, turning, and kicking. This has itself been a challenging task, and is work still in progress. High-level behaviors such as passing and maintaining formations are beginning to emerge, but at this stage, they play less of a role in determining the quality of play when compared to the proficiency of the agent's skills.

Figure 1. On the left is a screenshot of the Nao agent, and on the right a view of the soccer field during a 11 versus 11 game.
Figure 1. On the left is a screenshot of the Nao agent, and on the right a view of the soccer field during a 11 versus 11 game.

Agent Architecture

At intervals of 0.02 seconds, the agent receives sensory information from the environment. Every third cycle a visual sensor provides distances and angles to different objects on the field from the agent's camera, which is located in its head. It is relatively straightforward to build a world model by converting this information about the objects into Cartesian coordinates. This of course requires the robot to be able to localize itself for which we use a particle filter. In addition to the vision perceptor, our agent also uses its accelerometer readings to determine if it has fallen and employs its auditory channels for communication.

Once a world model is built, the agent's control module is invoked. Figure 2 provides a schematic view of the control architecture of our humanoid soccer agent.

At the lowest level, the humanoid is controlled by specifying torques to each of its joints. We implement this through PID controllers for each joint, which take as input the desired angle of the joint and compute the appropriate torque. Further, we use routines describing inverse kinematics for the arms and legs. Given a target position and pose for the foot or the hand, our inverse kinematics routine uses trigonometry to calculate the angles for the different joints along the arm or the leg to achieve the specified target, if at all possible. The PID control and inverse kinematics routines are used as primitives to describe the agent's skills, which are discussed in greater detail in Section 4.

Developing high-level strategy to coordinate the skills of the individual agents is work in progress. Given the limitations of the current set of skills, we employ a simple high-level behavior for 11 versus 11 games. We instruct the player closest to the ball to go to it while other field player agents assume set formational positions on the field computed using Delaunay triangulation [1] based on offset positions from the ball. Our predefined formations, as well as role assignment to determine which agent should go to which position in the formation, are described in [7]. The goalie is instructed to stand a little in front of our goal and try and stop the ball if it comes near.

Figure 2. Schematic view of UT Austin Villa agent control architecture.
Figure 2. Schematic view of UT Austin Villa agent control architecture.

Player Skills

Our plan for developing the humanoid agent consists of first developing a reliable set of skills, which can then be tied together by a module for high-level behavior. Our foremost concern is locomotion. Bipedal locomotion is a well-studied problem (for example, see Pratt [12] and Ramamoorthy and Kuipers [13]). However, it is hardly ever the case that approaches that work on one robot generalize in an easy and natural manner to others. Programming a bipedal walk for a robot demands careful consideration of the various constraints underlying it.

We experimented with several approaches to program a walk for the humanoid robot, including monitoring its center of mass, specifying trajectories in space for its feet, and using machine learning techniques to optimize a series of fixed key frame poses for the agent to cycle through in order to walk and turn in different directions [17]. After deciding that an omnidirectional walk gave us the best chance for quickly moving and turning, we chose to use a double linear inverted pendulum model based omnidirectional walk engine that was designed by our standard platform league team for use on the physical Nao robots. This walk engine, and associated optimization of parameters for the walk, are described in [8].

When invoking the kicking skill, the agent chooses from several different kicks as described in [11] and [9]. During kicking inverse kinematics is used to control the kicking foot such that it follows an appropriate trajectory through the ball. This trajectory is defined by set waypoints, ascertained through machine learning, relative to the ball along a cubic Hermite spline.

Two other useful skills for the robot are falling (for instance, by the goalie to block a ball) and rising from a fallen position. We programmed the fall by having the robot bend its knee, by virtue of which it would lose balance and fall. Our routine for rising is divided into stages. If fallen face down, the robot bends at the hips and stretches out its arms until it transfers weight to its feet, at which point it can stand up by straightening the hip angle. If fallen face up, the robot uses its arms to push its torso up, and then rocks its weight back to its feet before straightening its legs to stand. We optimized the rising movements of the robot for speed and stability as described in [9].

Videos of our agent's skills are available at a supplementary website.

Conclusions and Future Work

This paper has presented a high-level view of the architecture and design of the UT Austin Villa agent. A fairly comprehensive and in-depth description of our 2011 agent, which is the base for this year's agent, is given in a technical report [10].

The simulation of a humanoid robot opens up interesting problems for control, optimization, machine learning, and AI. The initial overhead for setting up the infrastructure is bound to be overtaken by the progress made through research on this important problem in the coming years. While the main emphasis thus far has been on getting a workable set of skills for the humanoid, it is conceivable that soon there will be a shift to higher level behaviors as well. A humanoid soccer league with scope for research at multiple layers in the architecture offers a unique challenge to the RoboCup community and augurs well for the future. There are numerous vistas that research in the 3D humanoid simulation league is yet to explore; these provide the inspiration and driving force behind UT Austin Villa's desire to participate in this league.

UT Austin Villa has been involved in the past in several research efforts involving RoboCup domains. Kohl and Stone [6] used policy gradient techniques to optimize the gait of an Aibo robot (4-legged league) for speed. Stone et al. [14] introduced Keepaway, a subtask in 2D simulation soccer [2, 5], as a test-bed for reinforcement learning, which has subsequently been researched extensively by others (for example, Taylor and Stone [15], Kalyanakrishnan et al. [4], and Taylor et al. [16]). Most recently the team has used the 3D simulation domain to explore learning walks for bipedal locomotion (MacAlpine et al. [8] and Farchy et al. [3]). We are keen to continue our research initiative in the 3D simulation league. Our initial focus for the 2014 competition will be on further optimizing our set of skills, to realize faster walks, more powerful and accurate kicks, etc. This line of research should be enhanced with the use of heterogeneous robot models for the 2014 competition. Banking on a reliable set of skills, we will seek to develop higher level behaviors such as passing and intercepting balls.

References

  1. H. Akiyama and I. Noda. Multi-agent positioning mechanism in the dynamic environment. In RoboCup 2007: Robot Soccer World Cup XI. Springer, 2008.
  2. M. Chen, E. Foroughi, F. Heintz, Z. Huang, S. Kapetanakis, K. Kostiadis, J. Kummeneje, I. Noda, O. Obst, P. Riley, T. Steffens, Y. Wang, and X. Yin. Users manual: RoboCup soccer server for soccer server version 7.07 and later. The RoboCup Federation, August 2002.
  3. A. Farchy, S. Barrett, P. MacAlpine, and P. Stone. Humanoid robots learning to walk faster: From the real world to simulation and back. In Proc. of 12th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS), May 2013.
  4. S. Kalyanakrishnan, Y. Liu, and P. Stone. Half field offense in RoboCup soccer: A multiagent reinforcement learning case study. Proceedings of the RoboCup International Symposium 2006, June 2006.
  5. H. Kitano, M. Asada, Y. Kuniyoshi, I. Noda, E. Osawa, and H. Matsubara. RoboCup: A challenge problem for AI. AI Magazine, 18(1):73–85, 1997.
  6. N. Kohl and P. Stone. Policy gradient reinforcement learning for fast quadrupedal locomotion. In Proceedings of the IEEE International Conference on Robotics and Automation, May 2004.
  7. P. MacAlpine, F. Barrera, and P. Stone. Positioning to win: A dynamic role assignment and formation positioning system. In X. Chen, P. Stone, L. E. Sucar, and T. V. der Zant, editors, RoboCup-2012: Robot Soccer World Cup XVI, Lecture Notes in Artificial Intelligence. Springer Verlag, Berlin, 2013.
  8. P. MacAlpine, S. Barrett, D. Urieli, V. Vu, and P. Stone. Design and optimization of an omnidirectional humanoid walk: A winning approach at the RoboCup 2011 3D simulation competition. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence (AAAI-12), July 2012.
  9. P. MacAlpine, N. Collins, A. Lopez-Mobilia, and P. Stone. UT Austin Villa: RoboCup 2012 3D simulation league champion. In X. Chen, P. Stone, L. E. Sucar, and T. V. der Zant, editors, RoboCup-2012: Robot Soccer World Cup XVI, Lecture Notes in Artificial Intelligence. Springer Verlag, Berlin, 2013.
  10. P. MacAlpine, D. Urieli, S. Barrett, S. Kalyanakrishnan, F. Barrera, A. Lopez-Mobilia, N. Ştiurcă, V. Vu, and P. Stone. UT Austin Villa 2011 3D Simulation Team report. Technical Report AI11-10, The University of Texas at Austin, Department of Computer Science, AI Laboratory, December 2011.
  11. P. MacAlpine, D. Urieli, S. Barrett, S. Kalyanakrishnan, F. Barrera, A. Lopez-Mobilia, N. Ştiurcă, V. Vu, and P. Stone. UT Austin Villa 2011: A champion agent in the RoboCup 3D soccer simulation competition. In Proc. of 11th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2012), June 2012.
  12. J. Pratt. Exploiting Inherent Robustness and Natural Dynamics in the Control of Bipedal Walking Robots. PhD thesis, Computer Science Department, Massachusetts Institute of Technology, Cambridge, Massachusetts, 2000.
  13. S. Ramamoorthy and B. Kuipers. Qualitative hybrid control of dynamic bipedal walking. Robotics: Science and Systems, II, 2007.
  14. P. Stone, R. S. Sutton, and G. Kuhlmann. Reinforcement learning for RoboCupsoccer keepaway. Adaptive Behavior, 13(3):165–188, 2005.
  15. M. E. Taylor and P. Stone. Behavior transfer for value-function-based reinforcement learning. pages 53–59, July 2005.
  16. M. E. Taylor, S. Whiteson, and P. Stone. Comparing evolutionary and temporal difference methods for reinforcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1321–28, July 2006.
  17. D. Urieli, P. MacAlpine, S. Kalyanakrishnan, Y. Bentor, and P. Stone. On optimizing interdependent skills: A case study in simulated 3D humanoid robot soccer. In Proc. of 10th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2011), May 2011.