Alle@Home 2017 Team Description Paper

Gordon Cheng, Karinne Ramirez-Amaro, Ilya Dianov, Pablo Lanillos, Rogelio Guadarrama, Emmanuel Dean, Emilia Lozinska, German Diez Valencia, Patrick Grzywok, Qiuhai Guo, Gasper Simonic, Xiao Wang

Institute for Cognitive Systems, Technical University of Munich

https://www.ics.ei.tum.de/robocup-athome/

Abstract This paper provides a description of the team Alle@Home from the Institute for Cognitive Systems from the Technical University of Munich. Our goal is to participate for the first time in the RoboCup@Home 2017 competition with the robot TIAGo as part of the open platform league. Our team has special focus on novel semantic reasoning methods which enables the recognition of human intentions and it also improves the perception, control, and navigation systems. Currently our robot can handle incomplete information, for example if the operator asks for a cola and the robot only finds a fanta, this found object will be given to the operator as an alternative drink. This solution is possible due to our proposed reasoning engine.

Introduction

The main research interests of our team include cognition, mobile manipulation, computer vision and human robot interaction (HRI). The team consists of Bachelor, Master and PhD students who are advised by postdoctoral researchers and the university's professor. The team serves as a means for students to integrate their academic project work into a well-functioning robot control software system in real scenarios.

The code implemented for the demonstration shown for the qualification video can be found in the following repository:

https://gitlab.lrz.de/Robocup atHome ICS/Challenges.git The link to our team qualification video is https://youtu.be/A45ZCIzzkww

Team Alle@Home

The Institute for Cognitive Systems (ICS) offers students the practical course called RoboCup@Home where the students can directly interact with the robot and face real problems while learning different methods. This helps to enhance the abilities of a robot. In the scope of these courses the students design, develop and test new software components and try them out in the robot. The practical courses are supervised by postdoctoral researchers as well as by PhD candidates. The current team is supervised by Gordon Cheng and is led by Karinne Ramirez-Amaro.

The intention of the new practical course is that each year new students participate and get engaged in the development of the software of service robots such as TIAGo. The students from Fig. 1 are the first generation of the new practical course and now form part of team Alle@Home, which intents to participate in the RoboCup@Home 2017 for the first time.

Focus of research

Our team mainly focuses on deploying and improving robot capabilities through a cognition layer. This layer, in combination with state-of-the-art perception and navigation techniques, gives the robot the ability to handle complex scenarios when incomplete information appears. To this end, different new skills are added to our new robot TIAGo (e.g. reasoning, object recognition, navigation, manipulation, speech recognition, etc.), under the control of a semantic inference engine. Therefore, our robot can have improved decision making, task learning abilities, and contextual navigation. This approach suits the challenges defined for the RoboCup@Home competition where the robot should decide what to do depending on the information perceived from the environment. For instance, if a user asks for orange juice, the robot infers that it should go to the kitchen and, in the case of not finding the requested juice, the robot decides to bring something similar to the operator. Furthermore, as a second goal the team is researching on new perception systems based on deep neural networks enhanced with reasoning.

Technology and Scientific Contribution

In the context of the RoboCup@Home competition, we have developed a framework that integrates different capabilities in the service robot TIAGo (see Fig. 2). These capabilities are reasoning and knowledge representations, object recognition, navigation, kinematic control, speech recognition and face detection. With the developed framework the robot TIAGo is able to learn and recognize the operator, receive a command from an operator, navigate to the inferred location where objects are most likely to be found, find the requested or a similar object among other objects, grasp the desired object while at the same time avoiding obstacles, and finally give back the object to the operator. In the next subsections more description of this capabilities are described.

Reasoning and Knowledge

The main benefit of using knowledge and reasoning engine is that our robot can handle incomplete information. The reasoning system focuses on modeling the environment with the knowledge base and improving TIAGo's performance by using logical inference. The main problems tackled by the cognition system are to infer new solutions while performing tasks, and storing knowledge acquired during execution. For this, description logic is used as the reasoner and an ontology is used for the knowledge representation. This enabled a distinction between general rules and specific facts learned during execution. Methods for creating specific facts have been implemented as a set of ROS services and Prolog predicates through the KnowRob (Knowledge for Robots) system. They provide the tools to create instances of objects, assert properties and infer some knowledge about the system. For example, to infer the possible storage places for objects, a Prolog predicate was created which retrieves the necessary information from the knowledge-base (see Fig. 3.d). It searches for the storage places for objects (e.g. refrigerator for food) and possible types of rooms where the objects can be stored (e.g. kitchen for refrigerator).

Object Recognition

As a first approach, we have integrated the package find_object which recognizes objects in 2D using SURF descriptors and by means of the depth information provided by the RGBD sensor, the 3D centroid on the optical camera frame is computed. Afterwards, the centroid is transformed into the base coordinate system, whose origin is defined as the centre of the bottom of the robot. We define the object pose by the centroid and a fixed rotation angle, see Fig. 3.a. Our next approach includes the integration of 3D descriptors as Fast Point Feature Histograms (FPFH) and deep learning techniques such as tensorflow. This implementation has been partially implemented in the robot to improve the robustness of the recognition of objects. However, their performance has not yet achieved enhancement over the baseline approach and we are still validating this approach. Furthermore, we have integrated face recognition and people detection using Histogram of Oriented Gradients (HOG) descriptors and Support Vector Machine (SVM) classifiers. Further enhancements are being pursued.

Object Manipulation

In order to compute the trajectory of the robot's arm to grasp the object we use Moveit! in combination with an octomap for collision avoidance. The depth data is used to build the octomap that encodes the spatial restrictions of the workspace. Then the Movit! planner computes a feasible trajectory. For this, we use the OMPL (Open Motion Planning Library) which is a free library for the planning of movements in MoveIt!. The goal of the sample-based motion planning is to find a collision-free path between the starting state of the robot and the target state. The search takes place within a physical workspace, the limitation of which is an obstacle for the robot. The state space of the robot consists of all its possible configurations within the working area. A single point in this space is a possible condition. The difficulty of solving the problem lies in the many degrees of freedom of the robot, which requires a high dimensionality of the state space. For example grasping something is considered as a redundant task for a 7 DoF arm in the robot TIAGo.

Navigation

We enhanced the standard ROS navigation stack. For building the map we use gMapping with Rao-Blackwellization and AMCL for localization. First, the robot should localize itself by rotating around to guess about where it is respect to the map of the room. For this, we use the AMCL (Adaptive Monte Carlo Localization) with laser scanner data. This navigation module has been improved by including the interaction with the reasoning system, then the robot can navigate to the new desired place where objects are most likely to be found. For example, if the user ask for a fanta, the robot should go to the kitchen, this information is be provided by the reasoning system. For the global and local planner we use the one provided by PAL. Our robot successfully avoids obstacles while navigating towards the goal. Currently, we are also integrating an optimized contextual navigation to provide prior knowledge about the environment to the robot in order to optimize its navigation. For instance, in order to look for the operator the robot can use its latest available information about the operator position to navigate through the regions with higher probability. This method can optimize a function that maximizes the probability of detecting the object being pursued. This kind of techniques are easy to combine with the cognitive layer as prior knowledge which can be constructed from acquired behaviors of the user to enable a contextual navigation system.

Speech Recognition

We have integrated the CMUSphinx toolkit which is a leading speech recognition toolkit for speech applications. We use this system to better communicate with the robot and make the interaction more natural.

Learning by Demonstration

Allowing robots to recognize activities through different sensors and re-using its previous experiences is a prominent way to program robots. For this, we propose a recognition method that is transferable toward different domains independently of the used input sources. One key component for such generalization is the definition of common representations. We propose a hierarchical approach to extract the meaning of demonstrations by means of symbolic and semantic representations. These symbolic representations are used to generate a semantic reasoning engine to transfer the obtained models among different domains. Our reasoning-based learning system allows robots to re-use their previous experiences to correctly segment and recognize new Kinesthetically demonstrated activities for different tasks. This module is under development and it is planned to be fully implemented for the competition to teach the robot new tasks on-demand.

Task Learning

With increasing complexity of the robot environment the acquisition of the new tasks can be a very time consuming process as a robot has to obtain a new task every time its environment changes. In order to improve and accelerate task acquisition, we introduce a new method to teach robots new tasks in a fast and efficient manner by extracting key structures from the demonstrated tasks and utilising contextual knowledge from previous experience. The extracted structure is represented as a directed graph, which allows to capture important relationships between task components and simplifies the search of known tasks. Additionally, we connect the obtained task graph with an ontology to enhance the generalization of our method to make it applicable across different domains. This module is also under development and it will be use for the competition.

Conclusion

This document presents the description of the technological and scientific contributions of our team Alle@Home from the Technological University of Munich. Our service robot TIAGo has different capabilities such as reasoning & knowledge, object recognition, face and person identification, navigation in dynamic environments, human-robot-interaction, speech recognition. We believe that the contributions of our team as part of the RoboCup@Home competition will be beneficial to the robotics community.

Robot TIAGo Hardware Description

TIAGo has the following hardware description:

• Base: Differential drive system, max speed 1 m/s, 2 DoF.
• Torso: Lift stroke (1 DoF), 35 cm.
• Arm: Motor current feedback, mounted on torso. 4 DOF, Maximum load: 2 kg. Reach 87 cm.
• Wrist: Force/Torque sensor, 3 DoF.
• Head: 2 DoF, RGB-D camera.
• Gripper: Parallel gripper, 2 DoF.
• External devices: Laser range-finder, sonars, IMU, stereo microphones.
• Robot dimensions: height: 1.10m-1.45m, Base footprint 54 cm.
• Robot weight: 72 Kg.
• Battery: 36 V, 20 Ah.
• On board computer: CPU Intel(R) Core(TM) i5- 4590S @3.00GHz, 4GB RAM, 60 GB.
• External laptop: CPU Intel(R) Core(TM) i7-4510U @2.00GHz, 8GB RAM, 250 GB SSD.

Robot's Software Description

The software that we use to control the robot TIAGo is as available in the following repository:

https://gitlab.lrz.de/Robocup atHome ICS/Challenges.git

Some of our implementations are based on the PAL TIAGo tutorials: https://github.com/pal-robotics

Here is a summary of the implemented software:

• Platform: Ubuntu 14.04 Operating System
• Reasoning and Knowledge: Description logics and ontology, refer to [1–3].
• Navigation, localization and mapping: ROS Navigator with Reasoning.
• Face and people recognition: Histogram of Oriented Gradients (HOG) descriptors and Support Vector Machine (SVM) classifiers.
• Speech recognition: CMUSphinx toolkit
• Speech generation: Text to Speech Acapela.
• Object recognition: Package find object.
• Arm control and hand coordination: ROS-control and Moveit! in combination with octomap.

References

1. Karinne Ramirez-Amaro, Michael Beetz, and Gordon Cheng. Transferring skills to humanoid robots by extracting semantic representations from observations of human activities. Artificial Intelligence. DOI:10.1016/j.artint.2015.08.009, 2015.
2. Karinne Ramirez-Amaro, Michael Beetz, and Gordon Cheng. Automatic Segmentation and Recognition of Human Activities from Observation based on Semantic Reasoning . In IEEE/RSJ IROS 2014. IEEE, Sept 2014.
3. Karinne Ramirez-Amaro, Michael Beetz, and Gordon Cheng. Understanding the intention of human activities through semantic perception: observation, understanding and execution on a humanoid robot. Advanced Robotics, 29(5):345–362, 2015.
4. llya Dianov, Karinne Ramirez-Amaro, Pablo Lanillos, Emmanuel Dean-Leon, Florian Bergner, and Gordon Cheng. Extracting general task structures to accelerate the learning of new tasks. In IEEE-RAS International Conference on Humanoid Robots 2016, Cancun, Mexico, Nov 2016. IEEE.
5. Ilya Dianov, Karinne Ramirez-Amaro, and Gordon Cheng. Generating Compact Models for Traffic Scenarios to Estimate Driver Behavior Using Semantic Reasoning. In IROS, International Conference on Intelligent Robots and Systems, pages 69–74, Hamburg, Sep 2015. IEEE.
6. Moritz Tenorth and Michael Beetz. KnowRob: A knowledge processing infrastructure for cognition-enabled robots. I. J. Robotic Res., 32(5):566–590, 2013.
7. Labbé, M. Find-Object. http://introlab.github.io/find-object, 2011. accessed 2017-03-10.
8. Ioan Alexandru Sucan, Mark Moll, and Lydia E. Kavraki. The Open Motion Planning Library. IEEE Robot. Automat. Mag., 19(4):72–82, 2012.
9. Pablo Lanillos, Eva Besada-Portas, Jose Antonio Lopez-Orozco, and Jesus Manuel de la Cruz. Minimum time search in uncertain dynamic domains with complex sensorial platforms. Sensors, 14(8):14131–14179, 2014.
10. Karinne Ramirez-Amaro, Emmanuel C. Dean-Leon, and Gordon Cheng. Robust semantic representations for inferring human co-manipulation activities even with different demonstration styles. In IEEE-RAS International Conference on Humanoid Robots, pages 1141–1146. IEEE, 2015.
11. Emmanuel Dean, Karinne Ramirez-Amaro, Florian Bergner, Ilya Dianov, Pablo Lanillos, and Gordon Cheng. Robotic technologies for fast deployment of industrial robot systems. In 42nd IEEE Industrial Electronics Conference (IEEE IECON2016). IEEE, October 2016.