JSK@Home: Team Description Paper for RoboCup@Home 2017

Hiroaki Yaguchi, Ba Tran, Muku Takeda, Kanae Kochigami, Zijia Li, Kazuhiro Sasabuchi, Yuki Furuta, Kotaro Nagahama, Kei Okada, Masayuki Inaba

JSK laboratory, Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan

http://www.jsk.t.u-tokyo.ac.jp

Abstract JSK@Home is associated by robotics researhers of the JSK laboratory. We have been working on many research about home assistant robot and developing EusLISP as the mother environment for all JSK robots. We also have focused on various robotics fields like robot control and recognition, and published many useful ROS packages as the jsk-ros-pkg. In the RoboCup@Home we will focus on extending our achieved approaches in the manipulation, robot motions, and perception area to the interaction field. Through this competition, we plan to evalute our robotic software system and to develop novel human robot interaction method to achieve robust home assistant robot.

1 Introduction

The JSK Labratory have been conducting extensive research on dealing with various problems of robot perception. grasping, and manipulation in household environments. Our mother environment is geometrical model based system based on EusLISP [1], and we have developed ROS plugin for EusLISP to communicate our own robot system and various ROS software components. We have been publishing our robot softwares as open source software on GitHub[2], including robot control, recognition, etc. In the RoboCup@Home league, we focus to evaluate Daily-assistant functions we have developed, and plan to integrate and improve these softwares more closely in relation to human-robot interaction.

2 Background

Background section covering previous works and research areas.

2.1 Fetch and carry task with large scale indoor navigation

We have achieved large-scale indoor navigation using semantic database [3]. The proposed method has object information as common-sense knowledge with relative places. Also the robot has multi layerd 2D geometrical map information of the whole building and can use an elevator to go to another floor, recognize buttons and display information of the elevator. We evaluated the system in the sandwich fetch and delivery task. In the task, robot has no direct information about sandwich instance but has information that sandwich is a food. When the robot is asked to deliver sandwich, the robot recognize it is the food and its relative location is "kitchen" and "sandwich shop". Firstly try to find a sandwich from kitchen, if it is not found then nextly the robot buy at the shop on the another floor using an elevator.

2.2 Grasping object and using tools

We have been studying a lot about grasping object and tool manipulation by humanoid robots. These studies have an eye on robots' cooperation with human. Motions are learned by robot itself rather than manipulating robot by human directly. The way to acquire motion is by mainly watching human motion and imitating them. For example, humanoid robots observe the grasp motion by human and then reproduce the motion by modifying them for their body in the study [4]. In the study [5], the robot learns how to grasp tools through human-robot handover. Our study of [6], shows operating an object which is contact with what robot has in hand, for example, a boiled egg on a food turner. From these experiences, we have learnt how to generate arm moving trajectory and grasping motions for robots, and also how to extract information about some usual tools from human behavior.

2.3 Perception for object manipulation

We have specifically conducted a case study of a life-sized robot that can perform various daily chores including kitchen tasks, clothes handling, and sweeping[7]. Detection of clothes for clothes handling tasks is challenging as they are higly deformable objects. Wrinkle features, descripted by Garbor filters, are proposed for the task of cloth detection. Nishino et al proposes a method to find strings of characters from natural scene image, that provide much information for robots to understand the environments[8]. It finds character candidates by taking particular note of closed contours in an image, and detects character strings by evaluating their size and line consistency. Nagahama et al[9] proposes a novel method to estimate a tool's function by evaluating the relational hierarchy between a tool and objects obtained by tracking overlapped regions among them, and their accompanying movements. This enables robots to understand functions of tools while observing a person manipulates them.

2.4 Human Robot Interaction

Our focus of reseach contains how to observe people in order to reflect their feeling and deduce what kind of assistance is required. In addition, we have been trying to expand the human robot interaction design towards children. Bainbridge et al. [10] studied a methodology using sensor data from a humanoid robot to interpret a person's feelings. She focused on the relationship between sensor data and a person's impression towards a robot after the handshake interaction. Issac et al. [11] designed the robot system for finding a person who gets lost. By focusing on the head pose and the frequency of its change by using Neural Network, the person's state of "lost" or "not lost" can be detected. Kochigami et al. [12] focused on the commercially available robot "Pepper" and specified what it can and cannot do. This study is a first step for the system which a robot can get helped by a child when it has a diffucult task and therefore they can do chores together.

3 Help-me-carry Task

In the RoboCup@Home, we plan to improve and extend our previous works to achieve a user-friendly home assistant robot in the daily-life environment. For qualification, we describe an implementation of the Help-me-carry task. The robot knows geometrical map and table position to put a baggage, but does not know car position. In the task, we focus on following 5 functions; Task Management, Navigation, Person Tracking, Speech Recognition, and Hand Over Manipulation.

3.1 Task Management

A state graph of the task is presented. We use SMACH[13] for task management and define following 4 states. When the robot fails in each states, the robot will retry the state until success.

Follow to car: Firstly the robot recognize "Follow me" and track person H1. the robot will stop with "Stop following" and add Car spot with "Here is the car" on the map.

Pick bag up: Secondly the robot takes a baggage from a person H1 through hand over, the robot detects succeeded to take a baggage using its sensors.

Deliver bag to home: Thirdly the robot moves to kitchen table and puts baggage on the table, note that the robot knows entire map of environment and position of the kitchen table.

Guide person to car: After putting a baggage, the robot asks "Help me carry" to another person H2 near the table and waits until reply "OK" from a person H2. then the robot moves back to Car. Finally the robot declares "Here is the Car" when robot arrives at Car spot.

3.2 Navigation

To manage map, we use jsk maps package[14], is extension of ROS 2d nav cooperating gazebo and EusLISP geometrical environment model. Using this package, the map can be created only in the simulation environment applying EusLISP geometrical scene model into gazebo simulator. Also map server has "Spot" as a symbol of location, the robot can move target position using "Spot" name. "Spot" symbols can be added or removed dynamically, in the help-me-carry task "Car" spot is added via speech recognition.

3.3 Person Tracking

The Follow-to-car state requires person tracking. We employ KCF tracking [15] of OpenCV3, using head mounted 3D camera. KCF tracking is enough robust for pose change of the person, the algorithm can track person whether with front or back side, also when a person moves to out of frame.

3.4 Speech Recognition

In the Help-me-carry task speech recognition is the most important function to accept commands, e.g. spot registration in the "Follow to car" state and wait for reply in the "Guide person to car" state. For speech recognition we employ SpeechRecognition of Python package[16]. It can use several recognition engines and in the task we choose Google Speech Recognition.

3.5 Hand Over Manipulation

As describe in previous section, we have studied about human-robot hand over [5]. According to this study, the robot measures weight using wrist force sensor to detect putting a baggage on the gripper in hand over task.

4 Conclusions

The JSK Laboratory have been conducting research on robot perception, manipulation, open software platforms to introduce robots into households to assist humans. We believe with all the experieces we have accumulated on the field, RoboCup@Home league will be the place for us to evaluate our research results, and also share experiences with other teams from around the world.

Team Description

Team Name: JSK@Home

Country: Japan

Affiliation: JSK laboratory, Graduate School of Information Science and Technology, The University of Tokyo

Team Leader Name: Hiroaki Yaguchi

Contact Information: [email protected]

Website: http://www.jsk.t.u-tokyo.ac.jp/~nagahama/robocup/wiki/

Qualification Video: https://youtu.be/lTHkpvEBHJo

Team Members:

Students: Ba Tran, Yuki Furuta, Kazuhiro Sasabuchi, Kanae Kochigami, Muku Takeda, Zijia Li.

Supervisors: Masayuki Inaba, Kei Okada, Hiroaki Yaguchi, Kotaro Nagahama.

Standard Platform: HSR Description

Base: Omnidirectional wheel

Torso: 1 DoF elevator

Single arm: 4 DoF

Gripper: 1 DoF parallel link

Neck: 2 DoF (Pan - Tilt)

Range sensor: 1 horizontal LRF on the base

Force sensor: 6 axis in the wrist

Camera: IR pattern projection 3D camera, Stereo camera and Wide-range camera on the head, monocular camera in the hand.

Microphone: Microphone array on the head

Robot's Software Description

For our robot we are using the following software:

Platform: ROS, EusLISP [1], and jsk-ros-pkg [2]

Task Management: SMACH[13]

Navigation, localization and mapping: jsk maps [14]

Person Tracking: KCF Tracking[15] of OpenCV3

Speech recognition: Python SpeechRecognition Package[16]

References

1. M. Inaba, K. Okada, T. Yoshikai, R. Hanai, K. Yamazaki, Y. Nakanish, H. Yaguchi, N. Hatao, J. Fujimoto, M. Kojima, S. Tokutsu, K. Yamamoto, Y. Kakiuchi, T. Maki, S. Nozawa, R. Ueda, and I. Mizuuchi, "Enhanced Mother Environment with Humanoid Specialization in IRT Robot Systems," in Robotics Research: The 14th International Symposium ISRR, C. Pradalier, R. Siegwart, and G. Hirzinger, Eds. Springer, 2011, pp. 379–396.
2. JSK, "jsk-ros-pkg," https://github.com/jsk-ros-pkg.
3. M. Saito, H. Chen, K. Okada, M. Inaba, L. Kunze, and M. Beetz, "Semantic object search in large-scale indoor environments," in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Workshop on Active Semantic Perception and Object Search in the Real World, San Francisco, CA, USA, September, 25–30 2011.
4. T. C. Kieu, K. Yamazaki, R. Hanai, K. Okada, and M. Inaba, "Grasp observation and reproduction by humanoid robots using color camera and 3d sensor," in Proceedings of the 2011 IEEE/SICE International Symposium on System Integration, 2011, pp. 808–813.
5. W. P. Chan, Y. Kakiuchi, K. Okada, and M. Inaba, "Determining proper grasp configurations for handovers through observation of object movement patterns and inter-object interactions during usage," in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2014, pp. 1355–1360.
6. K. Nagahama, K. Yamazaki, K. Okada, and M. Inaba, "Manipulation of multiple objects in close proximity based on visual hierarchical relationships," in Robotics and Automation (ICRA), 2013 IEEE International Conference on, 2013, pp. 1303–1310.
7. K. Yamazaki, R. Ueda, S. Nozawa, M. Kojima, K. Okada, K. Matsumoto, M. Ishikawa, I. Shimoyama, and M. Inaba, "Home-assistant robot for an aging society," Proceedings of the IEEE, vol. 100, no. 8, pp. 2429–2441, 2012.
8. T. Nishino, K. Yamazaki, and K. Okada, "Extraction of character string regions from scenery images based on contours and thickness of characters," in MVA, 2011.
9. K. Nagahama, K. Yamazaki, K. Okada, and M. Inaba, "Hierarchical estimation of multiple objects from proximity relationships arising from tool manipulation," in 2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2012). IEEE, 2012, pp. 666–673.
10. W. Bainbridge, S. Nozawa, R. Ueda, K. Okada, and M. Inaba, "A methodological outline and utility assessment of sensor-based biosignal measurement in humanrobot interaction," International Journal for Social Robotics, vol. 4, no. 3, pp. 303–316, 2012.
11. A. Isaac, H. Yaguchi, T. Yoshikai, K. Okada, and M. Inaba, "An actively altruistic mobile robot system for identifying someone who looks lost using head pose tracking," in Proceeding of the 2011 IEEE/SICE International Symposium on System Integration, 2011, pp. F1–3.
12. K. Kochigami, Y. Kakiuchi, K. Okada, and M. Inaba, "Study on the feasibility and the design to support child care for a child to do chores with an interactive robot pepper," in Proceedings of the XXIII IFHE (International Federation for Home Economics) World Congress 2016 (IFHE 2016), august 2016, pp. 42–43.
13. J. Bohren and S. Cousins, "The smach high-level executive [ros news]," IEEE Robotics & Automation Magazine, vol. 17, no. 4, pp. 18–20, 2010.
14. JSK, "jsk maps," https://github.com/jsk-ros-pkg/jsk_demos/tree/master/jsk_maps.
15. J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, "High-speed tracking with kernelized correlation filters," Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2015.
16. Uberi and bobsayshilol and arvindch and kevinismith and haas85 and DelightRun and maverickagm and kamushadenes and sbraden and tb0hdan and Thynix, "SpeechRecognition," https://pypi.python.org/pypi/SpeechRecognition/.