MRL Soccer 3D Team Description Paper

Siavash Aslani, Nima Shafii, S.Mohammad H.S.Javadi, Vahid Azizi, Omid Mohamad Nezami, Siavash Mahmoudian

Mechatronics Research Laboratory, Qazvin Azad University, Qazvin, Iran

Abstract This paper briefly describes efforts of MRL 3D Soccer Simulation Team during past year to develop humanoid locomotion skills for simulated robot in this league. Different approaches are followed and implemented in this way and Evolutionary Algorithms are selected among them as an effective solution to overcome complexity of implementation of such skills. However, the model that EA must work on is too important which should guarantee the convergence of learning. Two implemented models are described and experimental results are explained. Examining more machine learning techniques in this field, MRL is going to achieve a method which is fast and optimum to implement different humanoid locomotion skills and is discussed in this paper.

1 Introduction

Biped locomotion is an important subject in robotic science. Many researchers have focused on this field and there are many approaches suggested in this way. Most of these approaches are going to implement walk skill in humanoid robot, but no method exists yet that can generate gaits in robot as stable and fast as human's gaits.

There are two major approaches in bipedal walking researches; model-based and model free approaches [1][2]. In model-based approaches the designer constructs a physical model of the robot and then builds a controller for the model. Two well designed representations in this approach are Zero Moment Point (ZMP) and inverted pendulum [3].

In model-free approach, which is also called the embodiment or dynamics based, it is common to make use of the sensory information and associate it with motions. No physical model is used in this method that eases the implementation of the skill. There are three important researches in this field; Passive Dynamic Walking (PDW) [4], Central Pattern Generator (CPG) and Ballistic Walking. In PDW approaches, robot does not have any actuators and it walks slop just by using the gravity force. In CPG approaches, periodic functions provide base for changing gait's angles. The ballistic walking approach contains two phases for each leg, swing and support.

Section 2 introduces CPG and Hoapf Modeling which MRL is following to implement humanoid locomotion skills. Section 3 describes our implementations in genetic algorithms. The future works and conclusion are described in Section 4 either.

2. CPG and Hoapf Modeling

In human and animals, the neural system generates rhythmic signals that are sent to the musculo-skeletal system in order to produce torques on the different joints to achieve locomotion [5]. Locomotion can be described by the interaction of three elements:

1. Spinal Central Pattern Generators (CPG).
2. Sensory feedback.
3. Descending supra-spinal control.

In CPG reflexes are produced by the robot's feedback sensors and are used to manage perturbations and balance control. CPGs can be represented by different mathematical models such as oscillators, artificial neurons, vector fields and etc [6][7]. Each CPG usually is assigned to one degree of freedom (DOF) of a joint. Oscillator based CPGs use the concept of limit cycles which are very convenient in the case of locomotion since they can return to their stable state after a small perturbation and are almost not influenced by a change in the initial conditions.

Due to a perfect knowledge of the robot dynamics is not needed in CPG approaches, we've followed central pattern generation methods in MRL 3D Soccer Simulation. First we made an Open Loop CPG that interacts directly with the actuators but receives no feedback from the sensors. The main advantage of this model is the ability to easily control the amplitude and the frequency of the produced signal independently.

We are also following Hoapf oscillators approach to model the signal generator. The equation (1) describe this oscillator:

$$\dot{x} = (\mu - r^2)x - \omega y$$ $$\dot{y} = (\mu - r^2)y - \omega x$$ (1)

Where r = x 2 + y 2 , µ> 0 determines the amplitude of the signal and ω controls the frequency of the oscillator. This oscillator has a stable limit cycle with radius √µ and angular velocity ω rad/s (Fig.1).

Reducing parameters of the model in our approach, we use some methods of coupling in different joints. This means that the amplitude depends on the µi parameter but is also influenced by pervious joint Hoapf output signal. This phenomenon is explained by the fact that the two involved oscillators are in resonance since their respective frequencies are the same.

3. Genetics Algorithm

Genetics algorithm is the evolutionary computation that can find the best value of parameters for optimizing the fitness function. We had two implementations in genetic algorithms. In first implementation we used Fourier series as signal generator for each DOF. We evolved CPG parameters for 10 DOF including 4 hip joints, 2 knees and 4 ankle joints for left and right legs. We expanded equation (2) for N = 2. So six parameters were used to determine Fourier series for each DOF.

$$f(t) = \sum_{n=1}^{N} A_n \sin(n\omega t + \phi_n)$$ (2)

In this way number of genes in each individual exceeded to 60 numbers, but using phase shift between left and right leg it decreased to 30 genes. Fig.2. illustrates a chromosome used in this approach.

After a long generation and adapting fitness function the result was not hopeful at all and we turned to a new approach.

In second approach we divided robot's walking skill into 5 parts during time. Two first parts form first step that starts from standing state. Next 3 parts form second step which repeating it in a cycle leads to walking ahead (Fig.3.).

There is one parameter in each part for each DOF. In this way number of genes exceeds to 50 that's too much. But finding some relation among joints angular velocities during a step, we could decrease the number of genes to 27. One of these equations is mentioned below:

$$X_{23} = \left(\sum_{i=1}^{3} L_{2i}t_i + \sum_{i=1}^{2} x_{2i}t_i\right)/t_3$$ (3)

Where $X_{23}$ is angular velocity of left leg or right leg along first axis in 3rd part of step 2. And $L_{2i}$ is angular velocity of left leg during part 1 to 3 in second step. $t_i$ indicates duration of part i. The equation indicates that sum of movement of joint X should be zero at the end of step.

Table 1. Our genetics algorithm specification

Fig.4. illustrates convergence of genetic algorithm for first step made of 10 genes. After learning first step, we let the robot to learn both steps together how initial values for first 10 parameters were learned before. In this way robot learned both step together used to gait as Fig.3. But repeating second step in a cyclic manner was not stable at all and accumulative error leaded the robot to fall down in 3rd or 4th step.

At last, changing the fitness function, we made the robot to learn both steps again starting from initial population learned from previous try. In this approach the fitness was calculated regarding number of successful steps during walking skill which leaded the robot to learn walking ahead for at least 10 meters without any feedback and correction during time.

To make a more stable walking, we added a simple feedback control to walking skill and robot could walk ahead for a long distance.

4. Conclusion and Future Work

Central Pattern Generation is one of the most popular approaches for implementing locomotion skills in humanoid robots. Hoapf oscillators can provide a great presentation for signal generation how couples joints to each other which leads to a faster convergence. During implementation and testing the method, Hoapf modeling is better than other modeling methods, but the search space is as large as other modeling methods, so other types of evolutionary algorithms can be used to overcome such large search space. So we're going to have another implementation by PSO on same models to compare with genetic algorithm.

On the other hand, to ease the implementation different machine learning techniques, MRL is developing a learning framework and some other tools useful to convert idea to implementation as fast as possible. Fig.5. presents Pattern Builder tool to generate some middle-patterns as checkpoints for robot to guide it during learning procedure.

References

[1] K. Osuka. "From model based control to dynamics based control" 'explicit model' and 'implicit model' in robotics. ISCIE Journal Systems, Control and Information 43(2), 94 (1999). [2] K.Osuka. "Dynamics based control of mechanical systems". In Proceedings of 2001 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM '01), pp. [3] M. Vukobratovic,B. Borovac, and D. Surdilovic. "Zero-moment point proper interpretationand new applications". In Proceedings of the 2nd IEEE-RAS International Conference on Humanoid Robots, Humanoids 2001, pp. 237–244 (Waseda University, Tokyo, Japan, 2001). [4] McGeer T. "Passive dynamic walking." International Journal of Robotics Research. 9(2):62-82. 1990. [5]. A. J. Ijspeert. "Vertebrate locomotion. the handbook of brain theory and neural networks." 649:654. MIT Press. 2003. [6] A. Pikovsky, M. Rosenblum, J. Kurths. Synchronization, "A universal concept in non-linear sciences." Cambridge University Press. 2001. [7] K. Matsuoka. "Sustained oscillations generated by mutually inhibiting neurons with adaptation." Biol. Cybern.. 52:367-376. 1985