Arman Team - Research Proposal: Simulation Approach to Control Humanoid Agent Behavior

M.Ali Darvishdarab, M.Reza Shamshiri, Abolfazl Mahmoudnia, Mosalam Ebrahimi

Computer Science Department, Islamic Azad University Science and Research Branch, Tehran, Iran

Abstract According to the high degree of freedom of humanoid agent in the simulation server the dynamic stability needs to be well maintained. We defined some fundamental motions that can be selected using fuzzy decision rules. We designed a virtual neural network to control agent movement. The MATLAB-Simulink is used to simulate each behavior separately to extract real and accurate training data for the designed neural network.

1 Introduction

Obviously, the humanoid model of agent in the current server is much more challenging than that sphere model in previous versions of 3D server, simply because the dynamic stability of humanoid agents needs to be well maintained while the agent is performing walking, running, kicking or any other tasks. Furthermore, soccerplaying humanoid agents will have to handle the ball with both feet, and be robust enough to deal with possible challenges. A humanoid agent have high DOF (degreeof-freedom) to achieve various behaviors, such as locating, avoiding obstacles, shooting or passing the ball, getting up and etc.

2 Problem Space

To implement humanoid agent behaviors, the following fundamental motions are needed:

2.1 Keeping the humanoid balance (static or dynamic stability)

Static stability has to be considered when no torque is provided (moments is zero). In case of static stability, the only acting force is gravity, Kuffner et al.[1]. Dynamic stability, when robot actuators deliver torques, dynamic stability instead of static stability has to be considered. This concept builds upon the concept of zero moment point (ZMP), introduced by Vukobratovi´c [2]. Dynamic balance is particularly relevant during the stage of single support, i.e. when the agent stands on a single foot.

2.2 Walking patterns

An alternative to online motion planning is the generation of so-called offline patterns, i.e. walking primitives that can be used as building blocks in order to elaborate complex moves. The method proposed by Huang et al. [3] has the desirable property of generating gaits such that the hip motion is optimized to keep the ZMP in the center of the support region.

2.3 Gait optimization

Learning optimized gaits, a form of offline walking patterns, has been investigated by Hu et al. in [4]. They distinguish two phases in the gait pattern: a swing phase, when the robot stands on a single foot, and double support phase, when both feet are on the ground. Constraints and performance criteria are then introduced in order to cast the gait search as a constrained optimization problem. Constraints are due to the geometry of the agent, limits on the forces and velocities, and dynamic stability.

2.4 Getting up again

Falling down is typically seen as a catastrophic event in humanoid agents. In humanoid soccer, instead, certain tasks, like defending from an adversarial shot, hinge on the ability to promptly dive in the appropriate direction. As a consequence, the ability to quickly recover and get up again is also needed, because the game does not stop. Stuckler et al. investigated this specific task, an aspect usually ignored in related literature [5].

2.4 Kicking the ball

This behavior is unique feature of a humanoid agent. In fact in main stream humanoid research, contacts between the agent and the environment, besides walking, happen typically only for grasping. There appears to be few results available about planning a good kick. Kicking behavior can be programmed offline and then scheduled during the game when needed. An aspect that at the moment appears completely ignored is the impulsive collision with the ball.

3 Architecture

Humanoid agents are inherently redundant because they need to have real humanoid configurations. This degree of redundancy makes the development of an analytical solution for the inverse kinematics practically unapproachable [6]. For this reason, our proposed method considers the use of fuzzy decision rules with several artificial neural networks to solve the inverse kinematics of humanoid agent in new version of 3D-soccer server.

In this method we use a decision making system based on fuzzy rules in order to determine an appropriate behavior related to the conditions. Agent state, agent position, ball position, self and opponent team arrangement are some of important parameters that considered in fuzzy rules for decision making. After processing the conditions, an appropriate behavior that is selected by fuzzy rules will execute by its special section. In this phase we implement an artificial neural network for each behavior such as walking, running, swing, and etc. Implementation of these behaviors with a large neural network is too complex and the training phase is very time consuming process, moreover the extraction of realistic and accurate data for training set is impossible. In this case we want to divide behavior of a humanoid agent and its kinematics into some limited behaviors and then control each of them with their special neural networks. In this way the number of neurons of each layer and the training time will decrease and we have more efficient perception of the environment (efficient learning).

4 Implementation

In implementation phase we use Multi Layered Perception (MLP) neural network with extended back propagation training algorithm. MLP is a feed forward network and has many usages in robot control and parameter estimation. For example we design a 2-layer MLP-EBP with 6 neurons for the input layer, 100 neurons for hidden layer with logsig transfer function and 5 neurons with pureline transfer function for output layer of walking gait controller. Figure(1) shows the structure of agent behavior based on fuzzy rules and neural networks.

Since the robot should always remain stable and never fall, the learning set presented to the artificial neural network can be conveniently filtered to eliminate the undesired robot configurations and reduce the training process complexity. In this case we use MATLAB-Simulink to simulate each behavior separately with respect to COM and ZMP formulation and extract real and accurate parameters value for training set. Then we train our networks with respect to adaptation of learning rate in about 100 epochs.

4 Future Plane

In the near future we use Fuzzy Reinforcement Learning (FRL) [7, 8] to improve the robustness of decision making, against unexpected conditions.

References

1. J. Kuffner, S. Kagaami, K. Nishiwaki, M. Inaba, and H. Inoue, "Dynamically-stable motion planning for humanoid robots," Autonomous Robots, vol. 12, no. 1, pp. 105–118, 2002.
2. M. Vukobratovi´c and B. Borovac, "Zero-moment point thirty five years of its life," International Journal of Humanoid Robotics, vol. 1, no. 1, pp. 157–173, 2004.
3. Q. Huang, K. Yokoi, S. Kajita, K. Kaneko, H. Arai, N. Koyachi, and K. Tanie, "Planning walking patterns for a biped robot," IEEE Transactions on Robotics and Automation, vol. 17, no. 3, pp. 280–289, 2001.
4. L. Hu, C. Zhou, and Z. Sun, "Biped gait optimization using estimation of distribution algorithm," in Proceedings of the IEEE-RAS Interntaional Conference on Humanoid Roots, 2005, pp. 283–288.
5. J. St¨uckler, J. Schwenk, and S. Behnke, "Getting back on two feet: reliable standing-up routines for a humanoid robot," in Proceedings of the 9th International Conference on Intelligent Autonomous Systems, 2006.
6. J. Lope, R. Gonz´alez-Careaga,T. Zarraonandia and D. Maravall "Inverse Kinematics for Humanoid Robots using Artificial Neural Networks".
7. Zhou, C., Meng, Q.: Reinforcement Learning and Fuzzy Evaluative Feedback for a Biped Robot, in Proceedings of the 2000 IEEE International Conference on Robotics and Automation, San Francisko, USA, 2000, pp. 3829-3834.
8. Duško Katić, Vukobratović: Control Algorithm for Humanoid Walking Based on Fuzzy Reinforcement Learning, in 4th Serbian-Hungarian Joint Symposium on intelligent Systems ,2006.