Gemini in RoboCup-2004
Masayuki Ohta
Cyber Assist Research Center, National Institution of Advanced Industrial Science and Technology, AIST Tokyo Waterfront 2-41-6 Aomi Koto-ku Tokyo 135-0064 Japan
Abstract The purpose of Gemini is to test various machine learning methods. We challenged perceptual aliasing problems of reinforcement learning using an intermediate approach between model-based and model-free approaches. To improve efficiency by avoiding bootstrapping errors, we are trying a method in which agents skip the backup in perceptual aliasing states, dynamically adjusting backup length according to visited states.
Reinforcement Learning Approach
The purpose of Gemini is to test various machine learning methods. In the last year, we challenged perceptual aliasing problems of reinforcement learning. Our approach to this problem was an intermediate approach between modelbased approaches and model-free approaches. In our approach, we distinguished the perceptual aliasing states first, then, adopted stochastic selection only in the perceptual aliasing states and adopted greedy selection in the other states [1]. With this method, we could acquire policies which do not include infinite-loops with small computational cost, but the efficiency of the agents using the acquired policy is not good enough. The cause of this problem is in bootstrapping: the error of action-values spread to the other states widely when we adopt a reinforcement learning using bootstrapping. To solve this problem, we are trying a method in which agents skip the backup in perceptual aliasing states (Fig.1). In this method, the backup length dynamically changes according to the states that an agent visited. We have tested this method on a small grid-world example, and succeeded in improving the efficiency of the approach which we took last year[2]. However, the implementation to the SoccerServer domain has not finished yet.
References
- Masayuki Ohta, Yoichiro Kumada, Itsuki Noda. "Using Suitable Action Selection Rule in Reinforcement Learning", IEEE International Conference on Systems, Man, and Cybernetics, pages 4358-4363, 2003.
- Masayuki Ohta, Itsuki Noda. "Adjusting Backup-Length Automatically in Reinforcement Learning", The Second International Conference on Machine Learning and Cybernetics, pages 1624-1629, 2003.