ToBI - Team of Bielefeld A Human-Robot Interaction System for RoboCup@Home 2019

Sven Wachsmuth, Florian Lier, Leroy Rügemer, Sebastian Meyer zu Borgsen

Exzellenzcluster Cognitive Interaction Technology (CITEC), Bielefeld University, Inspiration 1, 33615 Bielefeld, Germany

http://www.cit-ec.de/de/ToBI · https://www.cit-ec.de/tobi · https://toolkit.cit-ec.uni-bielefeld.de/systems/versions/nao-minimal-nightly · https://toolkit.cit-ec.uni-bielefeld.de/systems/versions/robocup-champion-2016-2016-champion · http://wiki.ros.org/depth_image_proc · https://github.com/CentralLabFacilities · https://github.com/CentralLabFacilities/object_recognition · https://github.com/strands-project/strands_perception_people · https://github.com/CentralLabFacilities/bonsai · https://github.com/CentralLabFacilities/simple_robot_gaze

Abstract The Team of Bielefeld (ToBI) was founded in 2009. The RoboCup team's activities are embedded in a long-term research agenda towards human-robot interaction with laypersons in regular and smart home environments. The RoboCup@Home competition is an important benchmark and milestone for this goal in terms of robot capabilities as well as the system integration effort. In order to achieve a robust and stable system performance, we apply a systematic approach for reproducible robotic experimentation including automated task-driven regression tests. A second focus of research is the development of re-usable robot behaviors and robot skills. By re-usability we mean both, the re-use in different robot tasks as well as the re-use across different platforms. For RoboCup 2019, we have enhanced this approach for the standard platform Pepper which comes with certain requirements and limitations, like its own runtime and development ecosystem, limited computing resources onboard, or a limited range of sensor devices. We further introduced a simulation environment for the Pepper robot that is based on MORSE and allows to define additional artificial agents as human-like interaction partners. This is one of the key features for simulating complete RoboCup@Home tasks. Finally, we work on enhanced features of the Pepper platform starting with a 3D-printed mount for an additional laser scanner used for mapping, mixed-reality techniques for HRI, as well as bimanual object manipulation with bulky objects. In this paper, we will present a generic approach to these issues. System descriptions as well as build and deployment procedures are modeled in the Cognitive Interaction Toolkit. The overall framework inherently supports the idea of open research and offers direct access to reusable components and reproducible systems via a web-based catalog.

1 Introduction

The RoboCup@Home competition [1] aims at bringing robotic platforms to use in realistic domestic environments. Today's robotic systems obtain a big part of their abilities through the combination of different software components from different research areas. To be able to communicate with humans and interact with the environment, robots need to coordinate and dynamically configure their components in order to generate an appropriate overall robot behavior that fulfills parallel goals such as gathering scene information, achieving a task goal, communicating its internal status, and being always responsive to humans. This is especially relevant for complex scenarios in domestic settings.

The Team of Bielefeld (ToBI) was founded in 2009 and successfully participated in the RoboCup German Open as well as the RoboCup World Cup from 2009 to 2018. In 2016, the team ended first in several of the individual tests (Navigation, Person Recognition, GPSR, EE-GPSR, Restaurant) and, finally, won the global competition [2]. At RoboCup 2017, the team achieved the third place in the competition of the Open Platform League (OPL) and the seventh place in the Social Standard Platform League (SSPL). Finally, the team got the SSPL world champion at RoboCup 2018. Thus, our overall approach as been successfully ported to the Pepper platform which has to deal with (i) limited processing capacities on the platform and the low bandwith of the wireless connection to external computing resources, (ii) limited sensor capabilities, e.g., range and low resolution in space and time of the ultrasonic and laser sensors, (iii) its own ecosystem (NaoQi) which needs to be integrated with other ROSbased components. In the following sections, we will describe our approach to establish an improved development environment for the Pepper robot that allows to support the RoboCup activities as well as the more general research agenda on human-robot interaction.

Bielefeld University is involved in research on human-robot interaction for more than 20 years especially gaining experience in experimental studies with integrated robotic systems [3]. Within this research, strategies are utilized for guiding the focus of attention of human visitors in a museum's context [4]. For this purpose the robot needs to follow the gaze of humans as well as provide behaviors for object reference. Further strategies are explored in a project that combines service robots with smart environments [5], e.g. the management of the robot's attention in a multi-user dialogue [6]. An important baseline for any human-robot interaction experiment is that the reproducibility of robotic systems and their performance is critical to show the incremental progress – but that this is rarely achieved [7]. This applies to experimentation in robotics as well as to RoboCup. A Technical Description Paper (e.g. [8]) – as typically submitted to RoboCup competitions – is by far not sufficient to describe or even reproduce a robotic system with all its artifacts. The introduction of a systematic approach towards reproducible robotic experiments [9] has been turned out as a key factor to maximally stabilize basic capabilities like, e.g., navigation or person following. Together with appropriate simulation engines [10] it paves the way to an automated testing of complete RoboCup@Home tasks.

The Cognitive Interaction Toolkit provides a framework that allows to describe, deploy, and test systems independent of the underlying ecosystem. Thus, the concepts apply for ROS-based components and systems as well as for those defined with, e.g., NAOqi. Combined with an appropriate abstraction architecture, a re-usability of components and behaviors can be achieved across platforms. The CITK framework has already been applied to the Nao platform as well as to the Pepper platform for RoboCup 2017 and 2018. For the RoboCup@Home SSPL competition we further work on enhancing our simulation approach that allows to easily switch between the real hardware and a simulated environment including virtual sensors and actors. In order to keep our cross-platform approach, we utilized the MORSE Simulation framework [11] that additionally offers extended possibilities for modeling virtual human agents for testing human-robot interaction scenarios [12].

2 Robot Platforms

In 2016, ToBI participated in RoboCup@Home with the two service robots Biron and Floka, in 2017 with Biron and Pepper, in 2018 with Pepper. Figure 1 gives an overview of the three platforms (Pepper, TIAGo, Floka) that are currently used in our research. Although focussing on the Pepper for this year, we still aim at the development of platform independent as well as multiplatform robot capabilities. The Social Standard Platform Pepper (cf. Fig. 1(a)) is newly introduced to the RoboCup@Home competition in 2017. It features an omni-directional base, two ultrasonic and six laser sensors. Together with three obstacle detectors in his legs, these provide him with navigation and obstacle avoidance capabilities. Two RGB cameras, one 3D camera, and four directional microphones are placed in his head. It further possesses tactile sensors in his hands for social interaction. A tablet is mounted at the frontal body and allows the user to make choices or to visualize the internal state of the robot. In our setup we use an additional laptop as an external computing resource which is connected to the on-board computer of the Pepper via Wi-Fi. Because the onboard laser is quite short range, we developed a hardware mounting for an external laser sensor (Fig. 1(b)) that can be easily attached or removed. Thereby, the Pepper is enabled to build precise maps of the environment that can be used during competition for navigating with the limited onboard laser sensors. The robot platform TIAGO (cf. Fig. 1(c)) by PAL-Robotics is used in RoboCup@Home related research scenarios. It is equipped with laser sensors and sonars that allow nagivation and mapping, a pan-tilt head including rgb and depth cameras, and a compliant 7 DoF arm with a force/torque sensor and a parallel gripper. Our robot Floka (cf. Fig. 1(d)) is based on the Meka M1 Mobile Manipulator robotic-platform [2]. An omni-directional base with Holomni's caster-wheels and a lift-controlled torso enable navigating in complex environments. All three robot platforms are run with the same framework but slightly different robot skill implementations. This allows a transfer of robot behaviors on an abstract level.

3 System Architecture

Our service robots employ distributed systems with multiple clients sharing information over network. On these clients there are numerous software components written in different programming languages. Such heterogeneous systems require abstraction on several levels.

Figure 2 depicts a simplified overview of the system architecture used for the Pepper robot including an external processing resource — a single high performance laptop. In our architecture, the NAOqi framework still encapsulates hardware access to the robot, but we additionally managed to run ROS on the head PC of the Pepper. Our installation includes the entire ROS navigation stack and the depth processing pipeline for instance. This allows a further abstraction across different ecosystems and seamless integration. Software components from both worlds, NAOqi and ROS, can be flexibly deployed onboard or offboard the robot. Skills in the same ecosystem can communicate using ROS or native Qi messages, those in different ecosystems communicate through a ROS wrapper.

The computational resources on the robot's head PC are limited. Thus, only components that are time-critical, e.g. for safe and robust autonomous navigation, are deployed on the head PC, while other skills, like people perception, speech recognition, semantic scene analysis and behavior coordination, are running on the external laptop. In order to meet online processing requirements in certain robot behaviors, e.g. person following, depth and color images are streamed in a compressed format achieving frame rates of approximately 10Hz.

The robot behavior is coordinated using hierarchical state machines. The hierarchical structure consists of re-usable building blocks that refer to abstract sensors and actors, skills, and complete task behaviors. A typical abstract sensor would be a people perception, while a typical skill would be person following that already deals with certain interferences or robot failures like shortly loosing and, then, re-establishing a human operator. As far as possible, we re-use robot skills that already have been used on previous RoboCup@Home or related research systems [2], like Floka or TIAGo. However, this has certain limits if, e.g., a skill person following is based on dense, longer-range, high-frequency laser scans. The laser scans of the Pepper platform only achieve a frame rate of 6.66Hz with a very low resolution and reliable range. Therefore, we already merged the LIDAR with depth information from the camera located in the head of the robot. However, this requires that the robot looks down rather than looking up watching for people. Thus, this conflicts with other robot behaviors introducing new dependencies in the skill and behavior design of the robot. Abstracting skills from task behaviors still leads to a description of task-level state machines that are agnostic with regard to such considerations. The explicit definition of skills further allows to reason about them and track their success during the performance of the robot. Based on this, new elements had been introduced during the last years, like reporting on success and failure of tasks assigned to the robot in GSPR [2].

3.1 Development, Testing, and HRI Simulation

The software dependencies — from operating system dependencies to intercomponent relations — are completely modeled in the description of a system distribution which consists of a collection of so called recipes [9]. In order to foster reproducibility/traceability and potential software (component) re-use of the ToBI system, we provide a full specification of the 2016 system in our online catalog platform. The catalog provides detailed information about the soft- and hardware system including all utilized software components, as well as the facility to execute live system tests and experiments remotely. The MORSE simulation environment [11] allows to conduct human-robot interaction experiments and provides virtual sensors for the cameras and laser-range sensors (see Fig. 3(a)). The virtual image streams and laser scans are published on the equivalent ROS topics which are used by the real sensors. In Lier et al. [12], we show how to utilize this framework for an automated testing of a virtual human agent interfering with the navigation path of a robot (see Fig. 3(b)).

4 Facilitating HRI by mixed-reality techniques and bimanual object handovers

Further research is conducted with the Pepper platform towards two directions. The first direction explores how human-robot interaction can be facilitated by mixed-reality techniques (Fig 4(a)) [13, 14]. After the HoloLens is registered to a marker displayed on the screen of the Pepper platform, a coordinate transformation between the AR-device and the robot is established. This is used in the following to overlay the environment and robot seen by the user by further visualizations of the state (pose, map, laser scan) and intention (navigation path, grasp space, costmap) of the robot. This helps to get a better grasp of the robot's capabiities and enhances collaborative tasks, like teaching a robot behavior.

In Fig. 4(b), we show on-going work for bimanual manipulation with the Pepper. As illustrated in the upper part the manipulation strategy is robot agnostic. It applies the Task Constructor framework of MoveIt! [15]. Tasks are hierarchically represented using containers. Then grasps are selected by specific quality metrics that consider a bimanual pose for holding the object. First successful handovers where already shown at the RoboCup 2018 finals.

5 Conclusion

We have described the main features of the architecture and technical solution of the ToBI system for the RoboCup@Home Social Platform League (SSPL) 2019. Based on the already achieved development state and an analysis of the robot's performance at the last year's competitions, we further improved the software architecture and development cycle including task-driven testing using a simulation environment. The architecture allows to program and use robot skills across multiple ecosystems on both, internal and external computing resources of the robot. The incremental system development stages are completely reproducible by using the CITK environment. By focussing on the social standard platform, we are confident to further improve capabilities of the Pepper robot towards bimanual manipulation and enhanced HRI in order to allow furher application scenarios related to RoboCup@Home competitions.

6 Team information

Name of Team: Team of Bielefeld (ToBI)

Contact information: Sven Wachsmuth, Center of Excellence Cognitive Interaction Technology (CITEC), Bielefeld University, Inspiration 1, 33619 Bielefeld, Germany. Email: {swachsmu,semeyerz}@techfak.uni-bielefeld.de

Website: https://www.cit-ec.de/tobi

Team members: Sven Wachsmuth, Leroy Rügemer, Florian Lier, Sebastian Meyer zu Borgsen, Johannes Kummert, Luca Michael Lach, Kai Konen, Sarah Schröder, David Leins

Description of hardware: Pepper by Softbank Robotics (cf. section 2), external computing resource (Laptop) connected by WiFi

Description of software: Most of our software and configurations is open-source and can found at the Central Lab Facilities GitHub. Operating System: Ubuntu 16.04 LTS; NAOqi OS; OpenNao VM 2.4.3. Middleware: ROS Kinetic; RSB 0.16 [16]; NAOqi 2.5.5. SLAM: ROS Gmapping. Navigation: ROS planning pipeline. Object Recognition: use of Tensorflow (with re-trained top layers). People Detection: strands perception people. Behavior Control: BonSAI. Attention: Hierarchical Robot-Independent Gaze Arbitration. Speech Synthesis: Mary TTS. Speech Recognition: PocketSphinx with context dependent ASR.

References

1. Wachsmuth, S., Holz, D., Rudinac, M., Ruiz-del Solar, J.: RoboCup@Home benchmarking domestic service robots. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. AAAI'15, AAAI Press (2015) 4328–4329
2. Meyer zu Borgsen, S., Korthals, T., Lier, F., Wachsmuth, S. In: ToBI Team of Bielefeld: Enhancing Robot Behaviors and the Role of Multi-Robotics in RoboCup@Home. Volume 9776. Springer (2016)
3. Lohse, M., Siepmann, F., Wachsmuth, S.: A Modeling Framework for User-Driven Iterative Design of Autonomous Systems. International Journal of Social Robotics 6(1) (2014) 121–139
4. Pitsch, K., Wrede, S.: When a robot orients visitors to an exhibit. Referential practices and interactional dynamics in the real world. In: Ro-Man 2014. (2014) 36–42
5. Bernotat, J., Schiffhauer, B., Eyssel, F.A., Holthaus, P., Leichsenring, C., Richter, V., Pohling, M., Carlmeyer, B., Köster, N., Meyer zu Borgsen, S., Zorn, R., Engelmann, K.F., Lier, F., Schulz, S., Bröhl, R., Seibel, E., Hellwig, P., Cimiano, P., Kummert, F., Schlangen, D., Wagner, P., Hermann, T., Wachsmuth, S., Wrede, B., Wrede, S.: Welcome to the future – How naïve users intuitively address an intelligent robotics apartment. In: Proceedings of the 8th International Conference on Social Robotics (ICSR 2016). Volume 9979. (2016)
6. Richter, V., Carlmeyer, B., Lier, F., Meyer zu Borgsen, S., Kummert, F., Wachsmuth, S., Wrede, B.: Are you talking to me? Improving the robustness of dialogue systems in a multi party HRI scenario by incorporating gaze direction and lip movement of attendees. In: Proceedings of the Fourth International Conference on Human-agent Interaction, ACM Digital Library (2016)
7. Amigoni, F., Reggiani, M., Schiaffonati, V.: An insightful comparison between experiments in mobile robotics and in science. Auton. Robots 27(4) (November 2009) 313–325
8. Meyer zu Borgsen, S., Korthals, T., Wachsmuth, S.: ToBI-Team of Bielefeld The Human-Robot Interaction System for RoboCup@Home 2016. (2016)
9. Lier, F., Hanheide, M., Natale, L., Schulz, S., Weisz, J., Wachsmuth, S., Wrede, S.: Towards Automated System and Experiment Reproduction in Robotics. In Burgard, W., ed.: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE (2016)
10. Lier, F., Lütkebohle, I., Wachsmuth, S.: Towards Automated Execution and Evaluation of Simulated Prototype HRI Experiments. In: HRI '14 Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction, ACM (2014) 230–231
11. Lemaignan, S., Echeverria, G., Karg, M., Mainprice, J., Kirsch, A., Alami, R.: Human-robot interaction in the morse simulator. In: Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction, ACM (2012) 181–182
12. Lier, F., Lütkebohle, I., Wachsmuth, S.: Towards Automated Execution and Evaluation of Simulated Prototype HRI Experiments. In: HRI '14 Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction, ACM (2014) 230–231
13. Renner, P., Lier, F., Friese, F., Pfeiffer, T., Wachsmuth, S.: Facilitating HRI by Mixed Reality Techniques. In: HRI '18 Companion: 2018 ACM/IEEE International Conference on Human-Robot Interaction Companion, ACM/IEEE (2018)
14. Meyer zu Borgsen, S., Renner, P., Lier, F., Pfeiffer, T., Wachsmuth, S.: Improving Human-Robot Handover Research by Mixed Reality Techniques. In: VAM-HRI 2018. The Inaugural International Workshop on Virtual, Augmented and Mixed Reality for Human-Robot Interaction. (2018)
15. Görner, M., Haschke, R., Ritter, H., Zhang, J.: MoveIt! Task Constructor for task-level motion planning. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). (2018)
16. Wienke, J., Wrede, S.: A Middleware for Collaborative Research in Experimental Robotics. In: IEEE/SICE International Symposium on System Integration (SII2011), IEEE (2011) 1183–1190