RoboFEI@Home Team Description Paper for RoboCup@Home 2019

Bruno F. V. Perez, Douglas R. Meneghetti, Enrico Matiuci, Leonardo C. Neves, Fagner Pimentel, Gabriel S. Melo, João Victor M. Santos, Lucas I. Gazignato, Marina Y. Gonbata, Mateus G. Carvalho, Matheus V. Domingos, Rodrigo C. Techi, Thiago S. B. Meyer, William Y. Yaguiu, Flavio Tonidandel, Reinaldo Bianchi, Plinio T. Aquino Junior

FEI University Center, Sao Paulo, Brazil

http://robofei.aquinno.com/athome/

Abstract This team description paper (TDP) presents the Home Environment Robot Assistant, HERA, developed by the RoboFEI@Home team at FEI University Center, for the 2019 instance of the RoboCup@Home Competition. Throughout the text, the research goals and interests of the team will be presented, as well as the hardware and software stacks used in the development of the robot to solve the tasks of the @Home competition and some of the projects currently under development by the group. The project also motivates both undergraduate and graduate level research in the fields of computer vision, mechanics, embedded electronics, Internet of Things, artificial intelligence and human-robot interaction.

Introduction

Due to the concern regarding helping services, such as the need to assist humans in domestic and personal environments, assistive ways of using technology has increased. With the purpose of advancing the state-of-the-art in assistive robotics, the Robot Competition known as RoboCup@Home was created in 2006. Thus, to participate in the competition, an autonomous mobile robot is required.

The RoboFEI@Home team created HERA, a robot designed to perform human-robot interaction and cooperation tasks. At its core, HERA is based on the Mecanum Wheel robotic platform, having PeopleBot's chest-level extension to aid in the interaction with both humans as well as the environment.

HERA also counts with a series of sensors to aid in mapping and navigating the environment, as well as recognizing the human silhouette, individuals by their faces and objects. A gripper, designed and manufactured by the team, allows the robot to interact with the environment.

The rest of this TDP is organized as follows: section 2 presents RoboFEI@Home research focus and interests, as well as its history in other competitions. Section 3 describes the software stack used in the robot to solve the tasks of the @Home Competition. In section 4, we present the main projects under development by the RoboFEI@Home team, describe how they contribute to the RoboCup community, present their applications in the real world as well as current results. Finally, section 5 presents the conclusions and future works.

Research focus and interests

RoboFEI@Home focuses its research in the interaction between man and machine (computers, robots, autonomous cars or smart houses). Research in the field of human-machine interaction is crucial to the advancement of the state of the art, in which machines can act together with humans in their daily lives.

The project also makes heavy use of and contributes to the development of methodologies, techniques, models and algorithms in the following topics: adaptive interfaces, brain-computer interfaces; planning; intelligent residential and building automation; autonomous systems and the internet of things (IOT).

Team Achievements and Participations

RoboFEI@Home first participation in an @Home competition took place during Latin American Robotics Competition(LARC) 2015, in which the team ranked third. In 2016, the team made its debut in RoboCup, its first international competition, where it gained experience and maintained contact with other teams. The team also participated in LARC 2016 @Home competition in which we took first place. At LARC 2017 the team took first place again, being currently the Latin American champion in the @Home category.

The team also participated in the 16th International Machine Tools and Integrated Manufacturing Systems Trade Show, in 2017, and the 2050 Megatrends congress, in 2018. Also in 2018, the team participated in the RoboCup@Home competition in Canada, where the team took the seventh place in the competition and reinforced contact with other teams.

Description of the approach used to solve RoboCup@Home challenges

HERA uses ROS Kinetic Kame, released in May 2016. In the next sessions, will be present the vision, speech and navigation systems used in the robot.

Robot Vision

The task developed primarily for this TDP construction was to follow a specific person without the interference of the people in the surroundings. The robot performs a face recognition using Haar Cascade Algorithms from OpenCV, register a body by PCL ground-based RGBD detection and confirm the position using ROS leg detector package. The person asks for the robot to follow her/him. Lastly, the robot starts the process and only stops when commanded.

In order to perform object detection, two approaches were developed. The first one consists in the use of keypoint detection and description algorithms, such as scale-invariant feature transform (SIFT). The algorithm is applied to a small database of object pictures, which are pre-loaded in the robot's memory (roughly 4 to 6 pictures by object type). Then, the same keypoint detection and description algorithm is applied to frames of a live video feed and the keypoints from each frame are compared to the ones present in each image of the database using a feature matching algorithm, such as FLANN. A minimum number of matches is necessary for our algorithm to consider an object present in the scene.

This approach has the advantage of not requiring any model fitting nor a large image dataset, making it suitable for the competition environment. Its main disadvantage is in the high computational time necessary to process each frame, which grows linearly with the number of images in the robot's database.

Our second object detection method is based on the use of a convolutional neural network and the single-shot detector technique. With this approach, a dataset containing between 100 and 200 examples for each object is necessary. Object examples must be manually marked with bounding boxes, but the neural network is capable of greater generalization while reducing processing time. In this work, we use MobileNet v1 as our neural network architecture of choice, due to its design being focused for mobile hardware.

Objects are detected by any of the aforementioned methods using the video feed from the Kinect camera and are later placed in the 3D environment, with relation to the robot, by finding each object's corresponding location in the point cloud, provided by the Kinect infrared sensor.

Voice Recognition

The team decided to use the Google Speech Recognition API. For this, a ROS package was developed which operates by a set of APIs. These tools are online tools that works directly in Ubuntu.In addition, a comparison with generic phrases is made through the Hamming distance for the recognition of phrases variations.

Our team, developed a usage of this API by researching methods to make the code easier to adapt to a certain environment, creating a new use of words choices in the speech.

In this year competition the team is using MATRIX CreatorTM, a board with sensors, wireless communications and a FPGA. The main goal of using this board is to make a directional voice recognition, this way being able to recognize who is talking with the robot.

A Raspberry Pi connected to the MATRIX is used for communicating with the core of our robot. The Raspberry is responsible for reading the information of the many sensors in the board, then send this information for the main system.

Robot Navigation

When the robot is in an unknown location, it must do the environment, where it is located, mapping and at the same time defining its position in the space. This technique is known as Simultaneous Localization and Mapping (SLAM). In the navigation, the robot has the capacity to choose the best possible route and avoid possible obstacles. For this to happen, it determinate parameters where there is the slightest path error and the robot is constantly correcting it.

RoboFEI@Home Projects

(This section introduces the projects, content follows in subsections below)

Robot Localization with an Omnidirectional Base

For this year competition, is going to be use a new omnidirectional platform. The goal of replacing the old base of the robot consisted mainly of increasing the payload and the possibilities of hardware modifications to improve the performance of the robot's locomotion. In the last competition the team used the platform KUKA YouBot, which had a low payload and limited possibility of hardware modification. Then, was chosen to use this new base with the parts acquired in a external company and assembled by the team. Thus, the team achieved the improvement goals which were a greater payload, and interchangeability, facilitating a quick maintenance. The kinetics and control was totally made and implemented by the team. The new base can be seen in figure 3.

Hardware

The base is composed by four mecanum wheels that provides a high freedom, so the robot can move in any desired direction with the less motion possible needed, four 128 rpm motors, two Sabertooth motor controllers and one magnetic hall-effect encoder sensor for each motor. From the motors and their hall-effect type encoders is possible to receive the velocity and position information needed to send commands and control the movement of the base. The base is mainly run by an Arduino controller that has two fundamental functions: control the motors and read encoders data to send them to a computer running ROS in which, later, will compute the encoder values to integrate an odometry data.

Kinematic and control

The most basic form of localization is odometry, which is simply estimation of the vehicle pose by integrating estimates of its motion. The estimation of the robot pose is possible due to the constantly reading of the wheels encoders and the integration of this information.

When there is a path defined for the robot to navigate, a ROS package is responsible for sending velocity messages, so the path can be followed and, consequently, the destination can be reached. These commands are basically composed by two linear velocities (x, y) and one rotational velocity (th). With the mecanum wheels kinematics it's possible to determine the needed linear velocity for each wheel of the robot so the previous commanded velocities can be achieved. The Arduino controller then, is responsible for making the wheels turn at the calculated velocities. At the same time the controller is sending the commands to the wheels, it's also sending encoder sensor information to the ROS computer. Considering the encoder resolution and knowing the number of ticks for a full wheel revolution it's possible to do the reverse path where the encoder data for the wheels, based on the inverse mecanum wheels kinematics, is converted to ROS velocities (x, y and th). With the velocities in a known period it's possible to determine the robot pose variation and integrate the odometry information.

Upper set

The main objective of developing a new upper set for the robot was the need to keep the MATRIX Creator Board static in its position when the head start rotating. Besides that, the team desire a completely new appearance for the robot, making the human-robot interaction more friendly.

Mechanics

In the first part we prototyped a model and tested it using a CAD(Computer-Aided Design) program. The upper set was mainly made of aluminum and ABS plastic. The head structure is made of a 30 x 30 mm square section aluminum profile coupled to a shaft, a servo motor is responsible for rotating the entire structure, including an Apple tablet, a Microsoft Kinect and two RODE Microphones. The set can move 85° for both sides. For the angular movement of the Kinect sensor, a Dynamixel RX-24F servo motor was coupled to its stand, allowing the necessary movement of the sensor for people and object recognition. A 3D printer was used to create the stands for the servo motors and the sensors, because they are complex in shape and do not require a high mechanical resistance. To keep the MATRIX static, an axle is inserted through the hollow aluminum shaft and fixed in the head platform, this way keeping the board in top of the robot, allowing better sound caption.

Manipulator

This year the project of the currently manipulator being used was improved. The new manipulator still have the same number of degrees of freedom (DOF) contained in a human arm, with the purpose of obtaining a great similarity to real movements using the anthropomorphic principle.

Basing on that, an anatomy and human kinesiology study was initiate, more specifically on the skeleton of the free portion of the upper limbs, which are: arm, forearm, carpus, metacarpus. It was noticed that the main movements are extension and flexion.

With the geometry analysis of each component, the most appropriate manufacturing process, for the parts with more complex shapes, was a 3d printer and, for plain shape parts, a production process using aluminum, which results in a great resistance with less weight and a smaller dimension. For the manipulator movement, two Dynamixel XM430 and six XM540 actuators are used, which are distributed among shoulders, elbow and handle. All actuators have 360° of spin amplitude.

Kinematics and control

For the manipulator mobility some distinctive algorithms, which describe the freedom that it has in the space where it is located, are used. Manipulator extensors and junctions analysis methods that use direct and inverse kinematic were implemented, this way allowing to describe the behavior and state in the space. Together with an control algorithm, we can accomplish simple and even complex tasks, as the space information can be captured through the sensors and comprehended, so it can make decisions for new tasks and interactions using the manipulator.

Module for Human-Robot Adaptive Interaction Using Biological Signals

This scientific research focused on analyzing the environmental and biological variables of the user to create a more natural social interface between man and robot. Also encompassing the perception of human characteristics and its subsequent classification to enable a better understanding of the robot in the environment that it is located and how it can assist the user in their intentions and tasks at home.

The idea at the beginning is that the user has the equipment fixed in the body and so the robot can use the information coming from the device, in their actions. The equipment used in this project is the EMOTIV EPOC, considered to be flexible, versatile and more affordable compared to others of its kind and thus being ideal for the idea of wearable sensor.

Re-usability for other research groups

This module is available online and can be accessed by any team. This module can help other teams assist in the decision making of their robots so that the actions that the robot already executes depend on the reading of the module created and thus, positive actions, or that generate a certain happiness or neutrality, allow the interaction to continue Similarly. Unlike these, actions that leave the user with level of sadness detectable by the module are changed, thus allowing a more understandable and natural type of interaction, on the part of the robot.

Applicability in the real world

Research is conducted in an attempt to understand how humans react in the interactive process with robots, but it is difficult to capture the real feelings of the human during the construction of tasks shared between the robot and the people in the scenario. As the computer can understand our human reactions, feelings, emotions and social relationships, it can then better understand what actions to take in various everyday situations in an environment of human interaction at work, at home or in a hospital for example.

Results

For the validation of the module two main parts were analyzed, being one of them the statistical data of the training and the validation of the classifier used, and the other part the application of the module in real time in the humanrobot. The experiment was initially done with ten subjects and Table 1 shows the accuracy of the best result obtained from the training and validation of the implemented classifier, where it is possible to see the percentage of correctness of the classifier for each class. Table 1 shows the trained classes for emotional states, happiness, sadness and neutrality, respectively, represented by numbers 1, 2 and 3.

Table 1. Confusion matrix of the LDA classifier.

HERA Bot: Exploring the Frontiers of Communication

By exploiting these usability advantages, the project's purpose is to enable human-robot interaction to be raised to a non-face level, as a person can communicate through the smartphone and thus be able to perform any actions in person. For the development of the project, a python API was used to create a bot in the Telegram instant messaging application that communicates direct with the human-robot interaction actions of HERA. This bot establishes a robotuser communication path through the smartphone, such as a person on another smartphone, so that a remote conversation with the robot can be established.

Conclusion

The robotic platform custom made by our team, with an omnidirectional base, had a good performance in accomplishing the RoboCup competition tasks. There are still some improvements to be made, so the mobile robot will be able to accomplish all the tasks purposed in the competition. The first challenges have been successfully completed and the robot's autonomy is acceptable for the time required to perform the tasks. The speech recognition is currently improved, being able to even map the sound sources in the environment, and Deep Learning concept will be applied to the Object recognition and with this, researches and implementation of the remaining tasks expected in the RoboCup manual will occur.

Robot Technical Specifications

Hardware Description:

Base: Mecanum Wheel Robot platform.

• Sensors:
  • Hokuyo UTM-30LX-EW.
• Actuators:
  • Omnidirectional wheels.

Chest: Custom made extension.

• Sensors:
  • Emergency switch.
• Actuators:
  • 3D printed gripper 6 DOF.

Head: Apple Ipad 2" and Feedback Display.

• Sensors:
  • Microsoft Kinect;
  • Logitech c920 webcam;
  • 2 RODE VideoMic GO directional microphones;
  • MATRIX Creator Board;

Control: Intel Nuk i5 7500T CPU.

Software Description:

• OS: Ubuntu 16.04;
• Middleware: ROS Kinetic Kame;
• Localization/Navigation/Mapping: SLAM;
• Face detection: Haar cascades;
• Face recognition: LBP Algorithm;
• People Tracker: PCL and Leg detector;
• Gestures/movement recognition: OpenPose;
• Object recognition: SIFT + FLANN or MobileNet + SSD;
• Object manipulation: Inverse Kinematic implemented on ROS own packages;
• Speech recognition: Google Speech Recognition and other APIs;
• Speech synthesis: gTTs package.

References

1. ROS.org | Powering the world's robots. http://www.ros.org/. Accessed on: May, 10h 2017.
2. Matteo Munaro and Emanuele Menegatti. Fast rgb-d people tracking for service robots. Autonomous Robots, 37(3):227–242, 2014.
3. D.G. Lowe. Object recognition from local scale-invariant features. In Proceedings of the Seventh IEEE International Conference on Computer Vision, volume 2, pages 1150–1157, 1999.
4. Marius Muja and David G Lowe. Fast approximate nearest neighbors with automatic algorithm configuration. VISAPP (1), 2(331-340):2, 2009.
5. Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pages 21–37. Springer, 2016.
6. Tzutalin. Labelimg, 2015.
7. Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
8. Matrix creator board. https://www.matrix.one/products/creator. Accessed on: October, 26th 2018.