ToBI - Team of Bielefeld A Human-Robot Interaction System for RoboCup@Home 2018

Sven Wachsmuth, Florian Lier, Sebastian Meyer zu Borgsen

Exzellenzcluster Cognitive Interaction Technology (CITEC), Bielefeld University, Inspiration 1, 33615 Bielefeld, Germany

http://www.cit-ec.de/de/ToBI · https://www.cit-ec.de/tobi · https://toolkit.cit-ec.uni-bielefeld.de/systems/versions/nao-minimal-nightly · https://toolkit.cit-ec.uni-bielefeld.de/systems/versions/robocup-champion-2016-2016-champion · https://github.com/CentralLabFacilities · https://github.com/strands-project/strands_perception_people · https://github.com/CentralLabFacilities/simple_robot_gaze · http://wiki.ros.org/depth_image_proc

Abstract The Team of Bielefeld (ToBI) was founded in 2009. The RoboCup team's activities are embedded in a long-term research agenda towards human-robot interaction with laypersons in regular and smart home environments. The RoboCup@Home competition is an important benchmark and milestone for this goal in terms of robot capabilities as well as the system integration effort. In order to achieve a robust and stable system performance, we apply a systematic approach for reproducible robotic experimentation including automated tests. A second focus of research is the development of re-usable robot behaviors and robot skills. By re-usability we mean both, the re-use in different robot tasks as well as the re-use across different platforms. For RoboCup 2018, we plan to enhance this approach for the standard platform Pepper which comes with certain requirements and limitations, like its own runtime and development ecosystem, limited computing resources onboard, or a limited range of sensor devices. We further introduce a simulation environment for the Pepper robot that is based on MORSE and allows to define additional artificial agents as human-like interaction partners. This is one of the key features for simulating complete RoboCup@Home tasks. In this paper, we will present a generic approach to these issues. System descriptions as well as build and deployment procedures are modeled in the Cognitive Interaction Toolkit. The overall framework inherently supports the idea of open research and offers direct access to reusable components and reproducible systems via a web-based catalog.

1 Introduction

The RoboCup@Home competition [1] aims at bringing robotic platforms to use in realistic domestic environments. Today's robotic systems obtain a big part of their abilities through the combination of different software components from different research areas. To be able to communicate with humans and interact with the environment, robots need to coordinate and dynamically configure their components in order to generate an appropriate overall robot behavior that fulfills parallel goals such as gathering scene information, achieving a task goal, communicating its internal status, and being always responsive to humans. This is especially relevant for complex scenarios in domestic settings.

The Team of Bielefeld (ToBI) was founded in 2009 and successfully participated in the RoboCup German Open as well as the RoboCup World Cup from 2009 to 2017. In 2016, the team ended first in several of the individual tests (Navigation, Person Recognition, GPSR, EE-GPSR, Restaurant) and, finally, won the global competition [2]. At RoboCup 2017, the team achieved the third place in the competition of the Open Platform League (OPL) and the seventh place in the Social Standard Platform League (SSPL). A certain lesson learnt during the preparation for and the performance in the competition has been that our already established approach for cross-platform development of robotic behaviors and skills has not been sufficient in case of the Pepper robot. Thus, there was a large performance gap between both platforms used at the competition. The reasons are multi-faceted: (i) because of the limited processing capacities on the platform and the low bandwith of the wireless connection to external computing resources, some skills did not meet their performance requirements and needed to be redesigned using different communication patterns; (ii) because of a limited sensor range together with a low resolution in space and time of the ultrasonic and laser sensors, computational strategies for navigation and people following needed to be changed affecting behavior definitions on a higher strategic level. (iii) because of a parallel development of the simulation and testing environment, robot behaviors were not tested beforehand to a sufficient degree. (iv) with the same team members contributing to two competitions in parallel, the test preparation onsite has been sub-optimal. Late problems before or in the starting procedure of the new robot Pepper could not be fixed on time. In order to deal with all issues, the preparation time from platform delivery in March, 2017, to the competition in June, 2017, was too short. In the following sections, we will describe our approach to establish an improved development environment for the Pepper robot that allows to support the RoboCup activities as well as the more general research agenda on human-robot interaction.

Bielefeld University is involved in research on human-robot interaction for more than 20 years especially gaining experience in experimental studies with integrated robotic systems [3]. Within this research, strategies are utilized for guiding the focus of attention of human visitors in a museum's context [4]. For this purpose the robot needs to follow the gaze of humans as well as provide behaviors for object reference. Further strategies are explored in a project that combines service robots with smart environments [5], e.g. the management of the robot's attention in a multi-user dialogue [6]. An important baseline for any human-robot interaction experiment is that the reproducibility of robotic systems and their performance is critical to show the incremental progress – but that this is rarely achieved [7]. This applies to experimentation in robotics as well as to RoboCup. A Technical Description Paper (e.g. [8]) – as typically submitted to RoboCup competitions – is by far not sufficient to describe or even reproduce a robotic system with all its artifacts. The introduction of a systematic approach towards reproducible robotic experiments [9] has been turned out as a key factor to maximally stabilize basic capabilities like, e.g., navigation or person following. Together with appropriate simulation engines [10] it paves the way to an automated testing of complete RoboCup@Home tasks.

The Cognitive Interaction Toolkit provides a framework that allows to describe, deploy, and test systems independent of the underlying ecosystem. Thus, the concepts apply for ROS-based components and systems as well as for those defined with, e.g., NAOqi. Combined with an appropriate abstraction architecture, a re-usability of components and behaviors can be achieved across platforms. The CITK framework has already been applied to the Nao platform as well as to the Pepper platform for RoboCup 2017. For the RoboCup@Home SSPL competition we further work on appropriate simulation approaches that allow to easily switch between the real hardware and a simulated environment including virtual sensors and actors. In order to keep our cross-platform approach, we utilized the MORSE Simulation framework [11] that additionally offers extended possibilities for modeling virtual human agents for testing human-robot interaction scenarios [12].

2 Robot Platforms

In 2016, ToBI participated in RoboCup@Home with the two service robots Biron and Floka, in 2017 with Biron and Pepper. Figure 1 gives an overview of the three mentioned platforms. Although focussing on the Pepper for this year, we still aim at the development of platform independent as well as multi-platform robot capabilities.

The Social Standard Platform Pepper (cf. Fig. 1(a)) is newly introduced to the RoboCup@Home competition in 2017. It features an omni-directional base, two ultrasonic and six laser sensors. Together with three obstacle detectors in his legs, these provide him with navigation and obstacle avoidance capabilities. Two RGB cameras, one 3D camera, and four directional microphones are placed in his head. It further possesses tactile sensors in his hands for social interaction. A tablet is mounted at the frontal body and allows the user to make choices or to visualize the internal state of the robot. In our setup we use an additional laptop as an external computing resource which is connected to the on-board computer of the Pepper via Wi-Fi.

The robot platform Biron (cf. Fig. 1(b)) has been used in RoboCup@Home until last year. It is based on the research platform GuiaBot by adept/mobilerobots, customized and equipped with sensors that allow analysis of the current situation. The Biron platform has been continuously developed since 2001 and was used in RoboCup@Home from 2009 to 2017. Our robot Floka (cf. Fig. 1(c)) is based on the Meka M1 Mobile Manipulator robotic-platform [2]. An omni-directional base with Holomni's caster-wheels and a lift-controlled torso enable navigating in complex environments. For a detailed description of the robots Biron and Floka we refer to [2].

3 System Architecture

Our service robots employ distributed systems with multiple clients sharing information over network. On these clients there are numerous software components written in different programming languages. Such heterogeneous systems require abstraction on several levels.

Figure 2 depicts a simplified overview of the system architecture used for the Pepper robot including an external processing resource — a single high performance laptop. In our architecture, the NAOqi framework still encapsulates hardware access to the robot, but we additionally managed to run ROS on the head PC of the Pepper. Our installation includes the entire ROS navigation stack and the depth processing pipeline for instance. This allows a further abstraction across different ecosystems and seamless integration. Software components from both worlds, NAOqi and ROS, can be flexibly deployed onboard or offboard the robot. Skills in the same ecosystem can communicate using ROS or native Qi messages, those in different ecosystems communicate through a ROS wrapper.

The computational resources on the robot's head PC are limited. Thus, only components that are time-critical, e.g. for safe and robust autonomous navigation, are deployed on the head PC, while other skills, like people perception, speech recognition, semantic scene analysis and behavior coordination, are running on the external laptop. In order to meet online processing requirements in certain robot behaviors, e.g. person following, depth and color images are streamed in a compressed format achieving frame rates of approximately 10Hz.

The robot behavior is coordinated using hierarchical state machines in SMACH [13]. The hierarchical structure consists of re-usable building blocks that refer to abstract sensors and actors, skills, and complete task behaviors. A typical abstract sensor would be a people perception, while a typical skill would be person following that already deals with certain interferences or robot failures like shortly loosing and, then, re-establishing a human operator. As far as possible, we re-use robot skills that already have been used on previous RoboCup@Home systems [2], like Floka or Biron. However, this has certain limits if, e.g., a skill person following is based on dense, longer-range, highfrequency laser scans. The laser scans of the Pepper platform only achieve a frame rate of 6.66Hz with a very low resolution and reliable range. Therefore, we already merged the LIDAR with depth information from the camera located in the head of the robot. However, this requires that the robot looks down rather than looking up watching for people. Thus, this conflicts with other robot behaviors introducing new dependencies in the skill and behavior design of the robot. Abstracting skills from task behaviors still leads to a description of tasklevel state machines that are agnostic with regard to such considerations. The explicit definition of skills further allows to reason about them and track their success during the performance of the robot. Based on this, new elements had been introduced during the last years, like reporting on success and failure of tasks assigned to the robot in GSPR [2].

3.1 Development, Testing, and Deployment Toolchain

The software dependencies — from operating system dependencies to intercomponent relations — are completely modeled in the description of a system distribution which consists of a collection of so called recipes [9]. In order to foster reproducibility/traceability and potential software (component) re-use of the ToBI system, we provide a full specification of the 2016 system in our online catalog platform. The catalog provides detailed information about the soft- and hardware system including all utilized software components, as well as the facility to execute live system tests and experiments remotely. The MORSE simulation environment [11] allows to conduct human-robot interaction experiments and provides virtual sensors for the cameras and laser-range sensors (see Fig. 3(a)). The virtual image streams and laser scans are published on the equivalent ROS topics which are used by the real sensors. In Lier et al. [12], we show how to utilize this framework for an automated testing of a virtual human agent interfering with the navigation path of a robot (see Fig. 3(b)).

The development and deployment process by a researcher is illustrated in Fig. 4 (red numbers). It starts with the source code of her/his software components (Figure 4 (1)). These are often written in different programming languages and thus make use of diverse build environments. We address this issue by applying a generator-based solution that utilizes minimalistic template-based descriptions (recipes) of the different components that belong to a system distribution (Figure 4 (2)). Distribution files (Figure 4 (3)) are interpreted by a generator that creates build jobs on a continuous integration (CI) server. Additionally, a special build job is created that, if triggered, orchestrates the complete build and deployment process of the system. After all jobs are finished, the system is deployed (Figure 4 (4)) in the file system and is ready to use (Figure 4 (5)). Since setting up a CI server and the required plugins takes time and requires expert knowledge, we provide prepackaged installations for CITK users. Moreover, we recently introduced deployment of CITK-based systems using Linux containers, like Docker. System descriptions and their meta data, e.g., source code locations, wiki pages, issue tracker, current build status, experiment descriptions, and so forth are frequently synchronized to a web-based catalog that also implements the CITK data model – providing a global human readable and search-able platform which is a prerequisite for open research.

4 Conclusion

We have described the main features of the architecture and technical solution of the ToBI system for the RoboCup@Home Social Platform League (SSPL) 2018. Based on the already achieved development state and an analysis of the robot's performance at the last year's competitions, we improved the software architecture and development cycle in several aspects. The architecture allows to program and use robot skills across multiple ecosystems on both, internal and external computing resources of the robot. The simulation and testing environment has been improved in terms of virtual robot sensors as well as human-robot interaction scenarios. The incremental system development stages are completely reproducible by using the CITK environment. By focussing on the social standard platform, we are confident to further improve the capabilities of the Pepper robot to a significant degree utilizing the experiences from very sucessful RoboCup@Home competitions in 2009 to 2017.

5 Team information

Name of Team: Team of Bielefeld (ToBI)

Contact information: Sven Wachsmuth Center of Excellence Cognitive Interaction Technology (CITEC) Bielefeld University Inspiration 1, 33619 Bielefeld, Germany {swachsmu,semeyerz}@techfak.uni-bielefeld.de

Website: https://www.cit-ec.de/tobi

Team members: Sven Wachsmuth, Sebastian Meyer zu Borgsen, Florian Lier, Johannes Kummert, Dominik Sixt, Luca Michael Lach, Felix Friese, Kai Konen, Robert Feldhans, Sarah Schröder

Description of hardware:

• Pepper by Softbank Robotics (cf. section 2)
• external computing resource (Laptop) connected by WiFi

Description of software: Most of our software and configurations is open-source and can found at the Central Lab Facilities GitHub.

Operating System: Ubuntu 16.04 LTS; NAOqi OS; OpenNao VM 2.4.3 Middleware: ROS Kinetic; RSB 0.16 [14]; NAOqi 2.5.5 SLAM: ROS Gmapping Navigation: ROS planning pipeline Object Recognition: Classification Fusion (CLAFU) [15] People Detection: strands perception people Behavior Control: SMACH Attention: Hierarchical Robot-Independent Gaze Arbitration Speech Synthesis: Mary TTS Speech Recognition: PocketSphinx with context dependent ASR

References

1. Wachsmuth, S., Holz, D., Rudinac, M., Ruiz-del Solar, J.: RoboCup@Home benchmarking domestic service robots. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. AAAI'15, AAAI Press (2015) 4328–4329
2. Meyer zu Borgsen, S., Korthals, T., Lier, F., Wachsmuth, S. In: ToBI Team of Bielefeld: Enhancing Robot Behaviors and the Role of Multi-Robotics in RoboCup@Home. Volume 9776. Springer (2016)
3. Lohse, M., Siepmann, F., Wachsmuth, S.: A Modeling Framework for User-Driven Iterative Design of Autonomous Systems. International Journal of Social Robotics 6(1) (2014) 121–139
4. Pitsch, K., Wrede, S.: When a robot orients visitors to an exhibit. Referential practices and interactional dynamics in the real world. In: Ro-Man 2014. (2014) 36–42
5. Bernotat, J., Schiffhauer, B., Eyssel, F.A., Holthaus, P., Leichsenring, C., Richter, V., Pohling, M., Carlmeyer, B., Kster, N., Meyer zu Borgsen, S., Zorn, R., Engelmann, K.F., Lier, F., Schulz, S., Bröhl, R., Seibel, E., Hellwig, P., Cimiano, P., Kummert, F., Schlangen, D., Wagner, P., Hermann, T., Wachsmuth, S., Wrede, B., Wrede, S.: Welcome to the future – How naïve users intuitively address an intelligent robotics apartment. In: Proceedings of the 8th International Conference on Social Robotics (ICSR 2016). Volume 9979. (2016)
6. Richter, V., Carlmeyer, B., Lier, F., Meyer zu Borgsen, S., Kummert, F., Wachsmuth, S., Wrede, B.: Are you talking to me? Improving the robustness of dialogue systems in a multi party HRI scenario by incorporating gaze direction and lip movement of attendees. In: Proceedings of the Fourth International Conference on Human-agent Interaction, ACM Digital Library (2016)
7. Amigoni, F., Reggiani, M., Schiaffonati, V.: An insightful comparison between experiments in mobile robotics and in science. Auton. Robots 27(4) (November 2009) 313–325
8. Meyer zu Borgsen, S., Korthals, T., Wachsmuth, S.: ToBI-Team of Bielefeld The Human-Robot Interaction System for RoboCup@Home 2016. (2016)
9. Lier, F., Hanheide, M., Natale, L., Schulz, S., Weisz, J., Wachsmuth, S., Wrede, S.: Towards Automated System and Experiment Reproduction in Robotics. In Burgard, W., ed.: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE (2016)
10. Lier, F., Lütkebohle, I., Wachsmuth, S.: Towards Automated Execution and Evaluation of Simulated Prototype HRI Experiments. In: HRI '14 Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction, ACM (2014) 230–231
11. Lemaignan, S., Echeverria, G., Karg, M., Mainprice, J., Kirsch, A., Alami, R.: Human-robot interaction in the morse simulator. In: Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction, ACM (2012) 181–182
12. Lier, F., Lütkebohle, I., Wachsmuth, S.: Towards Automated Execution and Evaluation of Simulated Prototype HRI Experiments. In: HRI '14 Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction, ACM (2014) 230–231
13. Bohren, J., Cousins, S.: The SMACH High-Level Executive. IEEE Robotics & Automation Magazine 17 (2010) 18–20
14. Wienke, J., Wrede, S.: A Middleware for Collaborative Research in Experimental Robotics. In: IEEE/SICE International Symposium on System Integration (SII2011), IEEE (2011) 1183–1190
15. Roehlig, T.: Indoor room categorization using boosted 2d and 3d features. Master's thesis, University of Bielefeld, Citec, Bielefeld, Germany. (2014) Not published.