RoboFEI Team Description Paper for RoboCup@Home 2018
Marina Y. Gonbata, Leonardo C. Neves, Thiago S. B. Meyer, Rodrigo C. Techi, Pedro H. S. Domingues, Bruno F. V. Perez, Victor S. M. Schmiedl, Amanda M. Lima, William Y. Yaguiu, Matheus V. Domingos, Antonio C. O. Busnello, Kimberlyn K. G. Cardoso, João Victor C. M. Santos, Fagner Pimentel, Douglas De R. Meneghetti, Flávio Tonidandel, Plinio T. Aquino Junior
Centro Universitário FEI, São Paulo, Brazil
Abstract This team description paper (TDP) presents the robot Judith for the 2018 instance of the RoboCup@Home competition. Our robot was developed by the RoboFEI@Home team at the FEI University Center. Throughout this article will also be presented the objectives and interests of the group, the modules used in the development of the robot to solve the tasks of robocup@Home and some of the projects currently under development by the group. This project motivates research with students of scientific initiation, master and doctorate, in computer vision, mechanics, embedded electronics, IoT, artificial intelligence and humanrobot interaction.
Introduction
Due to the concern regarding helping services, such as the need to assist humans in domestic and personal environments, assistive ways of using technology has increased. With the purpose of advancing the state-of-the-art in assistive robotics, the Robot Competition known as RoboCup@Home was created in 2006. Thus, to participate in the RoboCup@Home competition, an autonomous mobile robot is required.
The RoboFEI@Home crew created Judith, a robot designed to perform humanrobot interaction and cooperation tasks. At its core, Judith is based on the KUKA YouBot platform, having PeopleBot's chest-level extension to aid in the interaction with humans, as well as the environment. Judith also counts with a series of sensors to aid in mapping and navigating the environment, as well as recognizing the human silhouette, individuals by their faces and objects. Finally, a gripper, designed and manufactured by the team, allows the robot to interact with the environment.
The rest of this TDP is organized as follows: In session 2 will be presented the focus of RoboFEI@Home research and its history in competitions. In session 3 the software development environment used in the robot to solve the tasks of the Robocup are briefly presented. In session 4 we present some of the projects under development by the RoboFei@Home team, their development, how they can contribute to the Robocup community, applications in the real world and some results. Finally, section 5 presents the conclusions and future works.
Focus of research and research interests
Intelligent technologies are becoming closer and closer to the human being. This project focuses on research involving the interaction between man and machines (computers, robots, autonomous cars or smart houses). These researches are therefore crucial to the development and advancement of these technologies to new heights where machines can make decisions or act together with the human being in their daily lives.
This project also focuses on the development of methodologies, techniques, models and algorithms in the following topics: Human-Computer Interaction; Adaptive Interface; Brain-Computer Interface; Planning; Intelligent Residential and Building Automation; Autonomous Systems; Internet of Things (IOT).
Team Achievements and Participations
The first RoboFEI@home team competition in the @Home category was in 2015 at the Latin American Robotics Competition (LARC), which took place in Uberlandia-Brazil where the team ranked third. In 2016, the team participated in Robocup in Germany, where it gained experience and maintained contact with other teams, being our first international participation. At the 2016 LARC, which took place in Recife-Brazil, the team won the competition. At LARC in 2017 the team took first place again, being currently the Latin American champion in the @home category. The team also participated in the FEIMAFE (16th International Machine Tools and Integrated Manufacturing Systems Trade Show) event and the 2050 Megatrends congress.
Description of the approach used to solve RoboCup@Home challenges
ROS (Robot Operating System) is a framework whose objective is to simplify software development for robots. ROS provides tools and libraries enabling the communication between a range of sensors and the control of actuators from different manufacturers. Judith uses ROS Indigo Igloo, released in July 2014 [1].
In the next sessions, will be present the vision, speech and navigations systems used in the robot.
Robot Vision
In the RoboFEI@Home project we utilize two input devices, Microsoft Kinect and a high definition camera, to achieve face recognition and object detection.
Each framework has a peculiarity such as, performance platform, objectives and even competition. In the RoboFEI@Home project the OpenNI [2],[3] was used, with the objective of accessing the main functions of the Kinect in several platforms.
The task developed primarily for this TDP construction was to follow a specific person without the interference of the people in the surroundings. The robot performs a face recognition using Haar Cascade and PCL Algorithms from OpenCV and register a body by pcl ground-based RGBD detection [4]. The Person asks for the robot to follow her/him. Lastly, the robot starts the process and only stops when commanded.
Object recognition is done using scale-invariant feature transform (SIFT) [5], an algorithm used in the detection and description of local features in images. A database of grayscale images is made available for the algorithm, each image containing one of the faces of an object that must be detected. The algorithm stores the keypoints extracted by SIFT for each image in-memory; when prompted to detect objects, it captures a frame from the camera currently being used, extracts keypoints from the frame and matches them against the keypoints of all images in the database. An object is considered detected when a minimum percentage of keypoints in one of its corresponding images is matched against the keypoints of the frame.
Besides representing a location in the image, each keypoint is represented by a feature vector, called the keypoint descriptor. SIFT uses the histogram of orientations around a keypoint as the keypoint's descriptor. Keypoint matching (also called feature matching) is done by calculating the distances between the vectors of keypoints.
Voice Recognition
The team decided to use the Windows speech Recognition alongside with Dragonfly package, which uses a virtual machine to run windows. These tools are offline tools that only communicate the Windows virtual machine and Ubuntu through a service created, based on server and client. Its recognizer was used along with our codes, resulting in an acceptable trustworthy, within 70% and 99% [6]. Our team, developed a usage of this API by researching methods to make the code easier to adapt to a certain environment, creating a new use of choices in the speech.
Robot Navigation
When the robot is in an unknown location, it must do the environment, where it is located, mapping and at the same time defining its position in the space. This technique is known as Simultaneous Localization and Mapping (SLAM). In the navigation, the robot has the capacity to choose the best possible route and avoid possible obstacles. For this to happen, it determinate parameters where there is the slightest path error and the robot is constantly correcting it.
Judith uses the laser sensor Hokuyo UTM-30LX-EW to create the environment 2D map where it is located, and find possible obstacles in the course. This device is used to scan areas, it has a 30 meters range and a 270° angular degree. Emitting an infrared light with a 0.36° pace, the number of scanned point is equal to 683 points. Through this it can locate new obstacles that weren't on the map.
RoboFEI@Home Projects
Manipulator
On the first part of development, some sketches were made with the main objective of creating a manipulator that would have the same number of degrees of freedom (DOF) contained in a human arm, with the purpose of obtaining a great similarity to real movements using the anthropomorphic principle.
Basing on that, an anatomy and human kinesiology study was initiate, more specifically on the skeleton of the free portion of the upper limbs, which are: arm, forearm, carpus, metacarpus. It was noticed that the main movements are extension and flexion.
In the project second phase, the prototype was built using Computer-Aided Design (CAD) tools, which allowed the study of the kinematics of movements and functional requirements, and if any member failed during the simulation, the project would be redesigned. This way, creating an iterative development process, with the intention of improving the prototype, as show on the figure 1.
With the geometry analysis of each component, the most appropriate manufacturing process, for the parts with more complex shapes, was a 3d printer and, for plain shape parts, a production process using aluminum, which results in a great resistance with less weight and a smaller dimension.
Mechanics
For the manipulator movement, one Dynamixel RX-24F actuators, two RX-28 actuators and three RX-64 actuator are used, which are distributed among shoulders (one RX-64 and one RX-28F), elbow (one RX-64) and handle (one RX-64, one RX-28 and one RX-24F). The actuator have 360° of spin amplitude. The RX-24F actuator is powered by 12V and both, the RX-28F and the RX-64 actuators by 18V. Using a movement system for more than one paddle flow, at the same time, reducing the number of servo motors.
Kinematics and control
For the manipulator mobility some distinctive algorithms, which describe the freedom that it has in the space where it is located, are used. Manipulator extensors and junctions analysis methods that use direct and inverse kinematic were implemented, this way allowing to describe the behavior and state in the space. Together with an control algorithm, we can accomplish simple and even complex tasks, as the space informations can be captured through the sensors and comprehended, so it can make decisions for new tasks and interactions using the manipulator. Our Dynamixel actuator provides a internal micro-controller that sends some informations about the state of motor. These informations, like the speed and goal error, are used into control algorithm providing more natural movements, since more complex movements can be executed. A easily way example to get more natural movements with kinematic control is through attributing different speeds for each DoF of the arm when it is going from the point A to the point B, these different speeds are proportional with the distances that each joint will course.
Module for Human-Robot Adaptive Interaction Using Biological Signals
This scientific research focused on analyzing the environmental and biological variables of the user to create a more natural social interface between man and robot. Also encompassing the perception of human characteristics and its subsequent classification to enable a better understanding of the robot in the environment that it is located and how it can assist the user in their intentions and tasks at home. The project should be included in the RoboCup@Home competition scenarios that are comprised of evidence to be solved by service robots that can work in homes, supermarkets, schools, hospitals and other public environments. In figure 2 is possible to see a scenary showing a human-robot interface that recognize emotions with a neural equipment.
The idea at the beginning is that the user has the equipment fixed in the body and so the robot can use the information coming from the device, in their actions. The equipment used in this project is the EMOTIV EPOC, considered to be flexible, versatile and more affordable compared to others of its kind and thus being ideal for the idea of wearable sensor.
Re-usability for other research groups
This module is available online [1] and can be accessed by any team. This module can help other teams assist in the decision making of their robots so that the actions that the robot already executes depend on the reading of the module created and thus, positive actions, or that generate a certain happiness or neutrality, allow the interaction to continue Similarly. Unlike these, actions that leave the user with level of sadness detectable by the module are changed, thus allowing a more understandable and natural type of interaction, on the part of the robot.
Applicability in the real world
Exploring the roots of human-robot interaction allows us to analyze and create a bond not only man-machine, but a social bond that is idealized by several researchers in this area. Research is conducted in an attempt to understand how humans react in the interactive process with robots, but it is difficult to capture the real feelings of the human during the construction of tasks shared between the robot and the people in the scenario. As the computer can understand our human reactions, feelings, emotions and social relationships, it can then better understand what actions to take in various everyday situations in an environment of human interaction at work, at home or in a hospital for example.
Results
For the validation of the module of emotional recognition developed, it is possible to analyze two main parts of the work, being one of them the statistical data of the training and the validation of the classifier used, and the other part the application of the module in real time in the human-robot. The experiment was initially done with ten subjects and Table 1 shows the accuracy of the best result obtained from the training and validation of the implemented classifier, where it is possible to see the percentage of correctness of the classifier for each class. Table 1 shows the trained classes for emotional states, happiness, sadness and neutrality, respectively, represented by numbers 1, 2 and 3.
Table 1. Accuracy (Class vs Class) of the LDA classifier.
| Cls vs Cls | 1 | 2 | 3 |
|---|---|---|---|
| 1 | 92.9% 5.8% | 1.3% | |
| 2 | 5.0% 93.8% 1.3% | ||
| 3 | 17.5% 17.5% 65.0% |
Judith Bot: Exploring the Frontiers of Communication
By exploiting these usability advantages, the project's purpose is to enable human-robot interaction to be raised to a non-face level, as a person can communicate through the smartphone and thus be able to perform any actions in person. For the development of the project, a python API was used to create a bot in the Telegram instant messaging application that communicates direct with the human-robot interaction actions of Judith. This bot establishes a robotuser communication path through the smartphone, such as a person on another smartphone, so that a conversation with the robot in a non-presential manner can be established.
Re-usability for other research groups
The Judith bot was made as a bundle in R.O.S., as it can be used by other robots changing only the specific features of the robot that use it.
Applicability in the real world
The bot developed allows to perform the same functions of commands and conversation that human-robot interaction face-to-face would have, therefore, functions of recognition of objects, people, voice, navigation actions, among others, are included in the non-presence interface. Examples of application are numerous, as the robot can talk to you at work, traveling or in another room and be assigned household chores such as checking items in a refrigerator, checking closed doors or windows in case of rain, sending pictures of the environment to verify people or even develop a conversation with the user.
Results
Judith bot was tested in the Latin American Robotics competition, where it performed household tasks and dialogue with the user in different rooms of a residence. Abilities such as recognition of people, objects and photos of environments, as well as operator recognition and navigation were evaluated in conjoint in a non-presence human-robot interaction and so a great help tool in several scenarios was noticed.
Conclusion
The robotic platform PeopleBot, with Kuka Youbot base, had a good performance in accomplishing the RoboCup@Home competition tasks. There are still some improvements to be made, so the mobile robot will be able to accomplish all the tasks purposed in the international competition. The first challenges have been successfully completed and the robot's autonomy is acceptable for the time required to perform the tasks. In the future, the speech recognition will be improved, being able to even map the sound sources in the environment, and Deep Learning concept will be aplied to the Object recognition and with this, researches and implementation of the remaining tasks expected in the RoboCup@Home manual will occur.
References
- ROS.org | Powering the world's robots. http://www.ros.org/. Accessed on: May, 10h 2017.
- OpenNI - Wikipedia, the free encyclopedia. https://en.wikipedia.org/wiki/OpenNI. Accessed on: August, 14th 2014.
- OpenNI 2 Downloads and Documentation | the structure sensor. http://structure.io/openni. Accessed on: August, 14th 2014.
- Matteo Munaro and Emanuele Menegatti. Fast rgb-d people tracking for service robots. Autonomous Robots, 37(3):227–242, 2014.
- D.G. Lowe. Object recognition from local scale-invariant features. In Proceedings of the Seventh IEEE International Conference on Computer Vision, volume 2, pages 1150–1157.
- Dragonfly speech recognition. http://pythonhosted.org/dragonfly/. Accessed on: September, 10th 2017.