RoboCup Rescue 2022 Team Description Paper Hector Darmstadt

Kevin Daun, Martin Oehler, Marius Schnaubelt, Stefan Fabian

Technical University of Darmstadt

www.teamhector.de · https://tdp.robocup.org/ · https://robocup-rescue.github.io/team_description_papers/

Abstract This paper describes the approach used by Team Hector Darmstadt for participation in the 2022 RoboCup Rescue Robot League competition. Participating in the RoboCup Rescue competition since 2009, the members of Team Hector Darmstadt focus on exploration of disaster sites using autonomous Unmanned Ground Vehicles (UGVs).

We provide an overview of the complete system used to solve the problem of reliably finding victims in harsh USAR environments. This includes hardware as well as software solutions and diverse topics like navigation, SLAM, human robot interaction and hazmat detection. In 2022, the team participates with a newly developed highly mobile robot platform. As a contribution to the RoboCup Rescue community, many parts of the used software have been released and documented as open source software for ROS.

I. INTRODUCTION

Team Hector Darmstadt (Heterogeneous Cooperating Team of Robots) has been established in late 2008. The team participated in RoboCup Rescue 2009 for the first time. Focusing on autonomy for rescue robot system, the team has a history of highly successful participation in the RoboCup Rescue Robot League competition. The "Best in Class Autonomy" Award was awarded to the team in the RoboCup German Open competitions from 2011-2015 and 2018-2021 and in the RoboCup world championships from 2012-2015 and in 2018-2019. The team also demonstrated that approaches leveraging a high level of autonomy are competitive with teleoperated robots. The team scored the first place in the overall scoring at RoboCup German Open 2011-2014 and 2021. Most notably, it won the world champion title at the RoboCup 2014 competition in Brazil. Furthermore, it participated successfully in RoboCup Worldwide 2021, winning the Awards for "Best in Class Dexterity" and "Best in Class Exploration and Mapping".

Several members of the team participated in the TOTAL ARGOS Challenge as part of Team ARGONAUTS. Here, many software technologies proven in the RoboCup Rescue Robot League were adapted for use in industrial inspection [1], with Team ARGONAUTS ultimately winning the ARGOS Challenge in 2017. Another success was achieved by winning the World Robot Summit Plant Disaster Prevention Challenge in 2018.

Contributing to an initiative within the RoboCup Rescue community to establish an open source framework for USAR robotics [2], the team has released many of the software modules used to achieve top scores at the RoboCup competition as open source software for ROS to facilitate progress and reduce the need for re-inventing the wheel [3].

A. Improvements over Previous Contributions

For increased mobility and dexterity, Team Hector developed an improved chassis with flippers carrying the modular vision box and sensor head from the previous robot, Jasmine.

B. Scientific Publications

Participating in RoboCup rescue plays an important role in our research by 1) motivating research topics, 2) evaluating our research at the competition, and 3) creating data sets for further research. Last year, we released several publications with advances related to the RoboCup.

We proposed a flexible framework for virtual projections to increase operator situation awareness, based on a novel method to fuse multiple cameras mounted anywhere on the robot [4]. Moreover, we introduce a complementary approach to improve scene understanding by fusing camera images and geometric 3D Lidar data to obtain a colorized point cloud. The implementation is available as open-source ROS packages. The fused point cloud is displayed in the robotics visualization tool RViz which we extended with an overlaid QML-based operator interface using the 2D human-robot interface tools [5] we have released as open-source ROS(2) package. The rough terrain at RoboCup Rescue induces fast roll- and pitch-motions on the robot, making state estimation a challenging problem. We propose HectorGrapher [6], a novel, time-continuous SLAM approach for robust and accurate pose estimation and mapping in real-time for challenging terrain. In our follow-up work [7] we extend SDF-based scan lidar registration for localization and navigation with radar under degraded visual conditions such as smoke or fog. For 2D SLAM in USAR environments, Hector SLAM [8] is a robust and efficient solution.

For supporting the robot operator while grasping arbitrary rigid objects, we developed a versatile grasping assistance method that combines an incrementally segmented 3D truncated SDF scene model with automated grasp pose detection [9].

For traversing rough terrain and obstacles as commonly found in USAR scenarios, we proposed a whole-body planner for autonomous mobile ground robots [10] and a highly efficient geometric pose prediction algorithm that is used to plan stability-based paths [11].

II. SYSTEM DESCRIPTION

This section describes the complete system for Asterix, a rescue robot platform developed by Team Hector Darmstadt for the 2022 RoboCup Rescue competition.

A. Hardware

For high mobility and dexterity combined with strong autonomous abilities, Team Hector developed Asterix, which consists of a chassis equipped with main tracks and flippers to carry the modular vision box, manipulator arm and sensor head. In the following, we briefly describe the key hardware components and sensors used for Asterix. A detailed description of the mechatronic system is available in [12].

1) Hardware Description of Asterix:

Asterix (see Figure 1) is designed to carry the modular vision box developed by Team Hector which is equipped with a continuously rotating lidar, IMU, two RGB-D cameras and the omnidirectional camera. A modular pan/tilt unit with an RGB-D camera and a thermal camera provides additional sensor information.

a) Flippers: For improved mobility, Asterix has four flippers actuated by two flipper motors. To minimize the sensor shadowing caused by the flippers for autonomous operation, one pair of flippers is completely foldable.

b) Manipulator Arm: For manipulation tasks, Asterix has a ROBOTIS Manipulator-Pro 6-DOF manipulator arm that is compactly foldable. On top of the gripper, an RGB-D camera, thermal camera and a color camera are mounted in order to perform inspection tasks.

c) LIDAR: The vehicle is equipped with a Velodyne VLP-16 Lidar attached to a continuously spinning mount. Using this setup, nearly complete coverage of all directions with highly accurate point cloud data is achieved.

d) Thermal Cameras: For victim and heat source detection, the robot is equipped with two Seek Thermal Compact thermal cameras. The first one is mounted on the pan/tilt unit and the second one is integrated into the gripper.

e) RGB-D Cameras: An Intel Realsense D435 RGB-D camera is used for object of interest and victim verification. This camera is mounted on the same pan/tilt unit as the thermal camera. Additionally, two Intel Realsense D455 RGB-D cameras are mounted on the lidar cage to provide depth data in close proximity of the robot.

f) 360 Degree Camera: An Insta360 Air camera is mounted on top of the Lidar cage and is used to acquire visual information from all directions. The robot operator uses a virtual pinhole projection to navigate the robot. Additionally, a 360 panorama projection is used for situational awareness. During detection tasks, the 360-image is used to recognize objects in all directions simultaneously without the need to move a camera.

g) Inertial Measurement Unit: To measure the attitude of the platform, the vehicle is equipped with a 9-DOF inertial sensor which measures accelerations and angular rates and estimates the orientation of the sensor.

h) Wheel/Track Encoders: To measure the translational and rotational speed of the vehicle, it is equipped with encoders measuring track motion. This odometry data is used for low-level speed control.

i) GPS receiver: As the vehicle can optionally be used outdoors too, it can be equipped with a GPS receiver. The position feedback provided by the SLAM system to the map is fused with information from GNSS in this case.

j) Locomotion & Flipper movement: The tracked locomotion of the robot is controlled by a Teensy 4.0 microcontroller, which is connected to the high-level system via rosserial. The microcontroller unit communicates with the four Elmo Whistle motor drivers using CAN to control the brushless AC servo motors (300 W). For controlling the position of the flippers, CUI AMT21 absolute encoders are coupled to the flipper shafts.

k) Computing: For the onboard autonomy of the robot, it has a quad-core and hexa-core Intel NUC equipped.

B. Software

1) SLAM: The Simultaneous Localization And Mapping (SLAM) problem is solved in 3D by using a modified variant of Google Cartographer [13] with a Truncated Signed Distance Function based map representation [6][14] optimized for use with spinning LIDAR data and rough terrain locomotion. Figure 2 shows an exemplary 3D map output.

Additionally, the map is fused with color information of the 360-camera to enable a better semantic understanding of the surroundings [4] (see Figure 3).

The map can be manually or automatically annotated with information about victims and other objects of interest. It can be saved in the GeoTIFF format using the hector geotiff package. This package is available and documented as open source software as part of the hector slam stack for ROS, which is widely used within the RoboCup Rescue League and beyond.

Software (continued) - Mapping, Image Projection, and Victim Detection

2) Mapping: Based on the data from our various depth sensors, including a spinning 3D LIDAR and multiple RGB-D depth cameras, and the localization provided by our SLAM approach, multiple maps are created. A 2.5D elevation map can be used for visualization and input for path planning algorithms. (Image-based) detections can be associated with a 3D pose using ray-casts in a 3D map of the environment, which is generated using a modified version of the octomap mapping package [15].

3) Image Projection: We use a single 360-camera mounted at the top of the robot for operator driving camera and panorama detection camera simultaneously. By flexibly creating views on-demand [4], we use two perspective projections for forward- and backwards-driving cameras and a Mercator projection for 360° object detection (see Figure 4). The viewing direction and zoom of the frontal projection can be controlled with the joystick to simulate a pan-tilt-zoom sensor head without any moving parts or actuation latency, increasing robustness in rough terrain.

4) Victim Detection: Finding human victims under difficult conditions of unstructured post-disaster environments is one of the main goals of RoboCup Rescue. Significant progress in visual object recognition and scene understanding allows us to apply state-of-the-art computer vision methods. To tackle this problem we use a multi-cue victim detection system supporting optical image cues like RGB, thermal and depth images. This complementary information can be used to increase reliability.

Once the detector has recognized a victim or other object of interest this detection is forwarded to the hector object tracker which keeps track of known objects and updates this model based on positive and negative evidence. The separation of object detection and modeling enables the flexible integration of different sensory sources for various classes of objects. The position and pose of each object is tracked using a Kalman Filter. The hector object tracker is the only interface between perception and control, e.g. for the creation or modification of tasks or the manipulation of model state due to operator interaction.

A comprehensive overview of our approach to semantic mapping using heterogeneous sensors such as thermal and visual cameras can be found in [16].

a) Thermal- and Depth-Based Victim Detection: In addition to visual victim detection, we use a thermal and also an RGB-D camera to verify vision-based hypotheses.

In most cases, images provided by the thermal camera are very helpful for identifying possible victim locations. As a drawback of a thermal camera, the thermal images often contain not only victims but also other warm objects, such as radiators or fire, so that thermal and visual recognition systems will deliver complementary information.

To further reduce false-positives, we use point clouds from the RGB-D camera to evaluate the environment of the victim hypotheses. False-positive victim hypotheses can be identified by the shape of the environment or by missing depth measurements at the victim location.

b) Hazmat Sign Detection: Hazmat symbols give important information to first-responders regarding the dangers they are facing at a disaster site. By correctly detecting and mapping hazmat signs, first-responders can prepare for toxic chemical substances, that might be leaking. We developed a hazmat sign detection based on SIFT-features and classification with templates. We run the detection on a panorama projection of the 360 camera for maximum coverage of the robot's surroundings. Detections are used as input for the hector object tracker as described above for the victim detection case.

Software (continued) - Motion Planning and Manipulation

5) Motion Planning: To handle the challenges posed by the rough terrain in the rescue arena, the path planning algorithm uses a real-time feasible pose prediction heuristic to plan stable paths based on the predicted contact points of the robots support polygon. Whereas traditional approaches are limited to binary traversability classification or handcrafted traversability estimations to be able to compute paths in real-time, our approach takes into account both the robot using the Unified Robot Description Format (URDF) and the raw 3D structure of the ground.

A 2.5D heightmap is used to represent the ground geometry. It is generated online from lidar and depth camera data. The two RGB-D cameras mounted on the cage fill blind spots of the lidar to cover the close proximity of the robot with a high update frequency.

The planned path is executed by a blind LQR-controller, which uses a kinematic motion model to minimize the expected path tracking error [17]. The resulting velocity commands are sent to the motor controllers.

More challenging obstacles like high steps or steep ramps might only be traversable by reconfiguring the robot kinematics while driving. Flipper joints can be used to keep stable contact with the ground and the manipulator arm shifts the robot's center of mass. We use an optimization-based motion planner to generate joint trajectories that enable the robot to autonomously overcome such obstacles [10].

6) Manipulation: We use the manipulation framework MoveIt as our interface to control the manipulator arm. We efficiently solve the inverse kinematics problem by formulating it as a nonlinear-least-squares optimization problem and solving it with Ceres. The arm can be controlled by either direct tele-operation or by planning to desired states. During tele-operation, the operator directly sets goal poses for the inverse kinematics using the gamepad. The closest solution is chosen for a smooth execution and self-collision constraints are obeyed. In a separate mode, the robot base can be moved while keeping the end-effector at a fixed position in world coordinates.

C. Communication

An Ubiquiti UAP-AC-M UniFi access point is used for high-bandwidth wireless communication. Both 2.4 GHz 802.11b/g/n or 5 GHz 802.11a/n/ac operation are possible. For distributing the network to multiple robot computers, the access points use relayd provided by the OpenWrt firmware to create a wireless bridge. The used SSID is "rrl hector darmstadt".

D. Human-Robot Interface

To enable seamless sliding autonomy control from pure teleoperation to full autonomy, an advanced user interface on top of existing ROS tools has been developed.

a) Mission Definition and Control: For defining autonomous or semi-autonomous behaviors, the FlexBE (Flexible Behavior Engine) approach developed within the scope of the DRC is used [18]. Using FlexBE, basic robot capabilities can be modeled via states and complex behaviors can be composed via the GUI by drag and drop. FlexBE supports selecting the desired autonomy level of the robot at runtime, making it very well suited for flexible control with an adjustable level of autonomy.

b) Monitoring and Human Supervision: As a basis for our operator interface, we use the open source robot visualization tool RViz which allows us to make use of the many integrated and publicly available visualizations of common robot sensor data. To augment the visualization of pre-processed 3D sensor data, we render a 2D operator interface on top of RViz that enables the control of the robot in a simple and intuitive fashion. For the visualization and control, we use techniques from popular video games to facilitate a quick on-boarding process for new operators who will find themselves familiar with the interaction design. Figure 5 shows a screen capture of this UI.

c) Teleoperation: In case supervisory control is not sufficient, the robot can be fully teleoperated using a game pad. In this case, the operator uses the aggregated world model generated from sensors onboard the robot and video streams to obtain situation awareness via the user interface. The operator can switch between a driving mode which controls base movement and flipper position and a manipulation mode for controlling arm and gripper. Common movements like folding/unfolding the arm or positioning the flippers for driving are easily accessible as separate behavior buttons. The motion is only executed as long as the button is pressed, so the operator is always in control. Our teleoperation software is available as open-source.

III. APPLICATION

This section describes the application of the Asterix system in practice.

A. Set-up and Break-Down

The standard system setup consists of one or more robots capable of autonomous or teleoperation via a laptop computer. All the control equipment easily fits into a standard backpack. The robots can be carried by two persons.

To start a mission, the robots and the laptop have to be switched on, and the operator can connect to the robots via WiFi.

B. Mission Strategy

As a focus of our research is reducing workload for operators and leveraging synergies between intelligent onboard systems and operators, pure teleoperation is only employed in case of failure of autonomous components. As during previous competition participation, autonomous operation is the desired control modality, possibly switching to a supervised autonomy mode for complex manipulation tasks that benefit from a human operator's superior cognitive and sense-making abilities.

C. Experiments

Robot systems are tested against subsets of standard NIST ASTM standard test methods that are reproduced in our lab. This includes a random maze, stairs and some of the proposed manipulation tasks. Importantly, testing of all system software components in simulation is a first class concept that is used to full extent by the team. Using the gazebo simulator, robots can be simulated within arbitrary disaster scenarios, allowing to evaluate performance and identify issues before costly and involved tests with the real system are performed. The hector nist arenas gazebo ROS package allows the fast and user-friendly creation of simulated disaster scenarios using elements of the NIST standard test arenas for response robots. Additionally, we use the test facilities of the German Rescue Robotics Center for larger scale experiments.

D. Application in the Field

We work in various research projects tightly together with first responders to bring our work to real world application. As part of the German Rescue Robotics Center [19], we investigate the application of ground robots in scenarios for 1) fire, 2) collapse, 3) flooding and 4) CBRN accidents. In the aftermath of the 2021 European floods, two team members and a robot were deployed in Erftstadt, Germany as part of the Robotic Task Force, led by the German Rescue Robotics Center. In February 2022, a large fire in Essen, Germany destroyed 39 apartments in a large apartment building. As the building was in danger of collapsing, large parts could not be accessed by human first responders anymore. Two team members deployed a rescue robot to explore the building and create 3D maps as part of the fire investigation.

IV. CONCLUSION

In this team description paper, we provide an outlook towards the RoboCup 2022 competition. We focus on highly reliable 3D mapping, autonomous rough terrain negotiation and automated perception of objects of interest. These capabilities have been already demonstrated in previous RoboCup competitions and will be further improved for participation in RoboCup 2022.

APPENDIX A: TEAM MEMBERS AND THEIR CONTRIBUTIONS

Many students and researchers at TU Darmstadt contribute to the team. The following list is in alphabetical order:

Team Members and Contributions

APPENDIX B: CAD DRAWINGS

CAD renderings of our robot platform Asterix are provided in Figure 6.

APPENDIX C: LISTS

An overview of the used hard- and software is provided in the Tables I, II, and III.

TABLE I: Asterix UGV

TABLE II: Operator Station

TABLE III: Software List

ACKNOWLEDGMENT

Research presented in this paper has been supported in parts by the German Federal Ministry of Education and Research (BMBF) within the subproject "Autonomous Assistance Functions for Ground Robots" of the collaborative A-DRZ project (grant no. 13N14861), by the LOEWE initiative (Hesse, Germany) within the emergenCITY center and by Nexplore within the AICO Collaboration Lab. We thank the department of computer science at Technical University of Darmstadt for the continuous support.

References

[1] S. Kohlbrecher and O. Von Stryk, "From robocup rescue to supervised autonomous mobile robots for remote inspection of industrial plants," KI-Kunstliche Intelligenz, vol. 30, no. 3-4, pp. 311–314, 2016. [2] S. Kohlbrecher, K. Petersen, G. Steinbauer, J. Maurer, P. Lepej, S. Uran, R. Ventura, C. Dornhege, A. Hertle, R. Sheh, and J. Pellenz, "Community-driven development of standard software modules for search and rescue robots," in Proceedings of the 10th IEEE International Symposium on Safety Security and Rescue Robotics (SSRR 2012), 2012. [3] S. Kohlbrecher, J. Meyer, T. Graber, K. Petersen, O. von Stryk, and U. Klingauf, "Hector open source modules for autonomous mapping and navigation with rescue robots," in RoboCup 2013: Robot World Cup XVII, ser. Lecture Notes in Computer Science, vol. 8371. Springer, 2014, pp. 624–631. [4] M. Oehler and O. von Stryk, "A flexible framework for virtual omnidirectional vision to improve operator situation awareness," in 2021 European Conference on Mobile Robots (ECMR), 2021, pp. 1–6. [5] S. Fabian and O. von Stryk, "Open-source tools for efficient ros and ros2-based 2d human-robot interface development," in 2021 European Conference on Mobile Robots (ECMR): Proceedings. IEEE, October 2021, event Title: 5th European Conference on Mobile Robots. [Online]. Available: http://tubiblio.ulb.tu-darmstadt.de/129739/ [6] K. Daun, M. Schnaubelt, S. Kohlbrecher, and O. von Stryk, "Hectorgrapher: Continuous-time lidar slam with multi-resolution signed distance function registration for challenging terrain," in 2021 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). IEEE, 2021, pp. 152–159. [7] M. Torchalla, M. Schnaubelt, K. Daun, and O. von Stryk, "Robust multisensor fusion for reliable mapping and navigation in degraded visual conditions," in 2021 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). IEEE, 2021, pp. 110–117. [8] S. Kohlbrecher, J. Meyer, T. Graber, K. Petersen, O. von Stryk, U. Klingauf, K. Petersen, A. Kleiner, O. von Stryk, S. Kohlbrecher et al., "Hector open source modules for autonomous mapping and navigation with rescue robots," Proc. RoboCup Symposium 2013, pp. 251–260, 2013. [9] M. Schnaubelt, S. Kohlbrecher, and O. von Stryk, "Autonomous assistance for versatile grasping with rescue robots," in Proceedings of the IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). IEEE, 2019, pp. 210–215. [10] M. Oehler, S. Kohlbrecher, and O. von Stryk, "Whole-body planning for obstacle traversal with autonomous mobile ground robots," in International Conference on Robotics in Alpe-Adria Danube Region. Springer, 2019, pp. 250–258. [11] S. Fabian, S. Kohlbrecher, and O. Von Stryk, "Pose prediction for mobile ground robots in uneven terrain based on difference of heightmaps," in 2020 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), 2020, pp. 49–56. [12] M. Schnaubelt, T. Ullrich, M. Torchalla, J. Diegelmann, M. Hoffmann, and O. von Stryk, "Entwicklung eines autonomiefokussierten hochmobilen bodenrobotersystems fur den katastrophenschutz," in Digital-Fachtagung VDI-MECHATRONIK 2021, T. Bertram, B. Corves, K. Janschek, and S. Rinderknecht, Eds. Universitats- und Landesbibliothek Darmstadt, 03 2021, pp. 20–25. [13] W. Hess, D. Kohler, H. Rapp, and D. Andor, "Real-time loop closure in 2d lidar slam," in 2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2016, pp. 1271–1278. [14] K. Daun, S. Kohlbrecher, J. Sturm, and O. von Stryk, "Large scale 2d laser slam using truncated signed distance functions," in 2019 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), Sep. 2019, pp. 222–228. [15] A. Hornung, K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard, "OctoMap: An efficient probabilistic 3D mapping framework based on octrees," Autonomous Robots, 2013, software available at http://octomap.github.com. [16] J. Meyer, P. Schnitzspan, S. Kohlbrecher, K. Petersen, O. Schwahn, M. Andriluka, U. Klingauf, S. Roth, B. Schiele, and O. von Stryk, "A Semantic World Model for Urban Search and Rescue Based on Heterogeneous Sensors," in RoboCup 2010: Robot Soccer World Cup XIV, ser. Lecture Notes in Computer Science, 2011, pp. 180–193. [17] A. Boyali, S. Mita, and V. John, "A tutorial on autonomous vehicle steering controller design, simulation and implementation," arXiv preprint arXiv:1803.03758, 2018. [18] P. Schillinger, S. Kohlbrecher, and O. von Stryk, "Human-robot collaborative high-level control with application to rescue robotics," in Proc. IEEE Int. Conf. on Robotics and Automation (ICRA), 2016, p. to appear. [19] I. Kruijff-Korbayova, R. Grafe, N. Heidemann, A. Berrang, C. Hussung, C. Willms, P. Fettke, M. Beul, J. Quenzel, D. Schleich et al., "German rescue robotics center (drz): A holistic approach for robotic systems assisting in emergency response," in 2021 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). IEEE, 2021, pp. 138–145. [20] R. B. Rusu and S. Cousins, "3D is here: Point Cloud Library (PCL)," in IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, May 9-13 2011. [21] P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features," in Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, vol. 1, 2001, pp. I–511–I–518 vol.1. [22] R. Lienhart and J. Maydt, "An extended set of haar-like features for rapid object detection," in Image Processing. 2002. Proceedings. 2002 International Conference on, vol. 1, 2002, pp. I–900–I–903 vol.1.