LyonTech, Team Description Paper RoboCup@Home 2019 Social Standard Platform League

Jacques Saraydaryan, Raphael Leber, Eric Lombardi, Laetitia Matignon, Fabrice Jumel, Olivier Simonin, Christian Wolf

CPE Lyon; INSA Lyon; UCBL Lyon 1 Univ.; CITI Lab., INRIA; LIRIS Lab., CNRS, Université de Lyon, Villeurbanne, France

https://robocup-lyontech.github.io/ · https://www.youtube.com/watch?v=xv1Da-CqZr4 · http://www.citilab.fr/ · https://team.inria.fr/chroma · https://team.inria.fr/chroma/en/plate-formes/

Abstract LyonTech consortium is ready and motivated for RoboCup @Home thematics and especially for the Social Standard Platform. We gather: i) highly qualified researchers in several areas of robotics (robot navigation, robot control, computer vision); ii) a fruitful collaboration between researchers and engineers; iii) past participations in the competition; iv) the integration of a large number of highly qualified students from different engineering schools (eq. Universities); v) expertise on ROS and Naoqi frameworks.

Introduction

La Doua - LyonTech is the main technology campus of Lyon, France. This campus is home to 25000 students, 1500 researchers and 1200 PhD students, spread out over a 100 hectare area. Different organizations inside the campus gathered to create the LyonTech team in order to participate to the RoboCup@Home challenge. The LyonTech team members belong to three teaching entities, two research laboratories, and mainly to the Chroma research team from INRIA:

• CPE Lyon, Engineering school, former RoboCup team in 2013 and 2016
• INSA Lyon, Engineering school, candidate for RoboCup 2016 organization
• University Claude Bernard Lyon 1 and its engineering school Polytech Lyon
• CITI Lab., Centre of Innovation in Telecommunications and Integration of Service (INRIA)
• LIRIS, Vision and information system laboratory (CNRS)

As we gather to create this team, we aim to use our synergy to contribute to the RoboCup@Home thematics with the Social Standard Platform.

This paper is organized as follows:

• Research and engineering interests
• Previous results and contributions to RoboCup and RoboCup@Home
• Our solutions for the SSPL
• Conclusion and references

Research and engineering competences

The LyonTech consortium consists of highly qualified researchers in computer science (AI, vision, navigation) working in research and teaching environments together with groups of highly skilled students. Research on robotics and application are well developed and structured. Two laboratories are involved in the LyonTech team.

CITI is an academic laboratory associated with INSA Lyon and INRIA. The CITI Laboratory develops research activities bringing together computer science, networking, and digital communications to address the challenging issues related to the development of IoT, fleet of connected vehicles and robots. The Lab. houses around 100 people, who are organized in 6 teams (see http://www.citilab.fr/).

Among these teams, Chroma is an INRIA team-project dedicated to humanaware navigation and multi-robot systems (https://team.inria.fr/chroma). Jacques Saraydaryan, Fabrice Jumel, Laetitia Matignon, Christian Wolf and Olivier Simonin are members of the Chroma team (led by Prof. O. Simonin), which researches focus on human-aware robot navigation and cooperation in dynamic environments.

LIRIS is a laboratory of the French Center of National Research (CNRS) with more than 300 researchers (140 research staff) from University of Lyon working in a variety of computer science fields: Computer Vision, Machine Learning and AI, geometry and modeling, data science, services, distributed systems, security, simulation, virtuality, computational sciences, interactions and cognition. The members involved in the LyonTech project are part of two different work groups, namely the Computer Vision Group (Imagine) and the Multi Agent Systems Group (SMA).

The RoboCup@Home challenge is an opportunity for team members to work on their specialties (image analysis, navigation, robot fleet management). It helps them to define use cases to drive research focus. For example the cases of robot waiter and tour guide robot are directly considered in our study benchmarks for navigation.

Previous results and contributions to RoboCup and RoboCup@Home

LyonTech is composed by former members of CPE Lyon team and by former candidates for RoboCup organization. We have participated the to the RoboCup competitions since 2013:

• "Lyon CPE" team : 3 rd place at RoboCup@Work, Joao Pessoa, Brazil, 2013
• "CPE Robot Forum" team : 15th place at RoboCup@Home OPL, Leipzig, Germany, 2016
• "LyonTech" team : 5 th place at RoboCup@Home SSPL, Montreal, CANADA, 2018

Additional contributions have been made to help the RoboCup organization and promotion:

• Lyon city and INSA candidated for the organization of the RoboCup, in 2016 (co-led by O. Simonin from Chroma/CITI team).
• Fabrice Jumel (CPE Lyon/CITI) is a RoboCup@Home evangelist for France, linked with the application of Bordeaux for RoboCup 2020 and the future French RoboCup regional committee. He was member of the TC of RoboCup@Home in 2017-2018. He is an OC member of RoboCup@Home SSPL since 2017. He was also an OC member at RoboCup@Home LARC in Recife in 2017.

Our solutions for the SSPL

Overview

The architecture of LyonTech's embedded AI software is shown in Figure 3. It contains modules which have been developed in different research groups of the consortium, completed by off-the shelf modules which tackle standard tasks, as well as engineering bricks interconnecting these modules. The scientific expertise of the consortium is broad and targets the needs of the competition:

Perception Our computer vision experts bring knowledge in gesture recognition [1], activity recognition from third person [2] and first person video [3], articulated pose estimation [4], semantic segmentation [5] and object recognition [6]. A large part of these methods are capable of running on real time and have been integrated in our platforms of mobile robots. Our combined work allows us to be aware of the objects present in a room, their locations, as well as the ongoing activities in this room.

Motion planning and Decision making Our expertise in robotics relates to motion planning in dynamic and uncertain environments, mapping, localization and decision-making for single and multi-robot systems. The work focuses on autonomous navigation in crowded environments (human-aware navigation) and in urban traffic (autonomous vehicles) for human assistance [7–9]. We also explore robot fleet cooperation for human scene observation [10, 11], 3D environment mapping, transport and service delivery [12]. We experiment and evaluate the models with Pepper humanoids, fleets of mobile robots (cf. Figure 2) and UAVs, and two equipped/autonomous cars (see https://team.inria.fr/chroma/en/plate-formes/).

Human-Robot Interactions We have been working for years on different interactions with robots (from teleoperation to multi-robots orchestration [13, 14]).

Integration The Ros Middleware is used to integrate components (customized packages and LyonTech packages), set of functional block are orchestrated through a General manager [15]. The section 4.2 detailed the set of tools and software used. In addition, the naoqi sdk, provided by softbank/aldebaran with the Pepper robot, gives a set of API that is mainly used for Robot and Human interactions (speech recognition, text to speech, robot behavior feedbacks). In order to highlight the robot activity, the Pepper tablet gives visual feedbacks (javascript framework).

The Robot Navigation Manager is in charge of localizing the robot and allowing dynamic navigation (obstacles avoidance). The analysis of the robot environment is performed by Object Detection and recognition modules, mainly deep neural networks developed inhouse [6] or off-the-shelf modules like YOLO 9000 [16]. Labeled object positions are provided to other blocks. All human robot interactions are managed by the Robot Human interaction block embedded in the robot. The robot also maintains a knowledge database about its environment (humans, objects and points of interest positions). Finally, the general manager block works like an orchestrator and gives order to other blocks in order to achieve scenarios.

Tools and contributions

Our solution is composed of a mix of existing and customized solutions, as well as full contributions.

Tools

• Pepper navigation: Ros Navigation Stack, we reused the Ros Navigation Stack with configuration modification including: navigation optimization (mainly on local planner), costmap customization (including set of sensors like 3d. sensors)
• Pepper User Interaction: With Naoqi sdk, we use the different functionalities to interact with users including dialogue and tablet interactions.
• Pepper Perception:
  • Darknet Yolo 2: we use the deep neural network framework Darnet-Net[17] with Yolo2 network, we build a procedure allowing use to fast capture images a retrain the network for our usage.
  • OpenPose: the OpenPose solution is used to detect human and estimate their position, we create an additional software to estimate different human pose (stand, sit,...) and distance.

Contributions

• General Orchestrator* [15] [18]: we developed a General Orchestrator allowing the robot to coordinate its capacities (navigation, perception, interaction) and make decision on defined scenario.
• Navigation Manager* [15] [19]: this functionality provide a set of navigation strategies depending on the observed context. Regarding to the context (lots of people, complex environment, large free space zone), the robot changes it way of navigation.
• Perception: in order to interact with people we developed a set of tools that accumulate knowledge about observed humans.
  • Face recognition* [20]: a solution was developed to automatically catch face and learn it for future detection.
  • Color detection* [21]: we extract main colors of a given picture (e.g t-shirt, trousers) based on HSV format and K-mean clustering helping us to track people.
  • Pose detection* [22]: based on the OpenPose data, we build a pose extractor gives us the estimate pose (stand, sit, lying down, left or right arm up,...)

Conclusion

We gave an overview of the approach which will be used by the LyonTech team to target the SSPL RoboCup@Home competition, including the different AI modules developed in the different research groups of the consortium. Fig. 4 and the video submitted with this paper present a scenario illustrating our ongoing work.

We aim to achieve better performances in Sidney, not only by improving the stage-one-like features, but also by keeping the vision of a more "general purpose" robot. We believe in the following strengths of the LyonTech consortium: i) highly qualified researchers in several areas of robotics which are vital for this competition (robot navigation, robot control, computer vision); ii) a fruitful collaboration between researchers and engineers; iii) past participations in the competition which allowed us to gain valuable experiences; iv) the integration of a large number of highly qualified students from different engineering schools (eq. Universities).

References

1. N. Neverova, C. Wolf, G.W. Taylor, and F. Nebout. Moddrop: adaptive multimodal gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 38(8):1692–1706, 2016.
2. F. Baradel, C. Wolf, J. Mille, and G.W. Taylor. Glimpse clouds: Human activity recognition from unstructured feature points. In CVPR, 2018.
3. F. Baradel, N. Neverova, C. Wolf, J. Mille, and G. Mori. Object level visual reasoning in videos. In ECCV, 2018.
4. N. Neverova, C. Wolf, G.W. Taylor, and F. Nebout. Hand pose estimation through weakly-supervised learning of a rich intermediate representation. To appear in Computer Vision and Image Understanding (CVIU), 2017.
5. D. Fourure, R. Emonet, E. Fromont, D. Muselet, A. Trémeau, and C. Wolf. Residual conv-deconv grid network for semantic segmentation. In British Machine Vision Conference (BMVC), 2017.
6. B. Moysset, J. Louradour, and C. Wolf. Learning to detect and localize many objects from few examples. IJDAR, 2018.
7. F. Jumel, J. Saraydaryan, and O. Simonin. Mapping likelihood of encountering humans: application to path planning in crowded environment. In The European Conference on Mobile Robotics (ECMR), Proceedings of ECMR 2017, Paris, France, September 2017.
8. M. Barbier, C. Laugier, O. Simonin, and J. Ibanez-Guzman. Classification of Drivers Manoeuvre for Road Intersection Crossing with Synthetic and Real Data. In 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, United States, June 2017.
9. M. Andries, O. Simonin, and F. Charpillet. Localisation of humans, objects and robots interacting on load-sensing floors. IEEE Sensors Journal, PP(99):12, 2015.
10. L. Matignon and O. Simonin. Multi-robot simultaneous coverage and mapping of complex scene - comparison of different strategies. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS, pages = 559–567, year = 2018,.
11. L. Matignon, S. D'Alu, and O. Simonin. Multi-robot human scene observation based on hybrid metric-topological mapping. In European Conference on Mobile Robotics (ECMR), 2017.
12. J. Saraydaryan, F. Jumel, and O. Simonin. Robots delivering services to moving people : Individual vs. group patrolling strategies. In IEEE ARSO, 2015.
13. L. Sevrin, N. Noury, N. Abouchi, F. Jumel, B. Massot, and J. Saraydaryan. Preliminary results on algorithms for multi-kinect trajectory fusion in a living lab. In IRBM, 2015.
14. E.Nauer A. Cordier, E. Gaillard. Man-machine collaboration to acquire adaptation knowledge for a case-based reasoning system. In ACM DL, editor, WWW 2012 – SWCS'12 Workshop, pages 1113–1120, Lyon, France, April 2012. ACM.
15. Jumel, Fabrice, Saraydaryan, Jacques, Leber, Raphael, Matignon, Laëtitia, Lombardi, Eric, and Wolf, Christian and Simonin, Olivier. Context Aware Robot Architecture, Application to the RoboCup@Home Challenge. In RoboCup symposium, pages 1–12, Montreal, Canada, June 2018.
16. J. Redmon and A. Farhadi. Yolo9000: Better, faster, stronger. In CVPR, 2017.
17. Joseph Redmon. Darknet: Open source neural networks in c. http://pjreddie. com/darknet/, 2013–2016.
18. Jacques Saraydaryan. General manager, https://github.com/jacquessaraydaryan/robocup pepper-general mng. GIT, 2018.
19. Jacques Saraydaryan. Navigation manager, https://github.com/jacquessaraydaryan/robocup pepper-navigation mng/tree/master/navigation manager. GIT, 2018.
20. Jacques Saraydaryan. Ros face recognition, https://github.com/jacquessaraydaryan/ros face recognition. GIT, 2018.
21. Jacques Saraydaryan. Ros main color detection, https://github.com/jacquessaraydaryan/ros color detection. GIT, 2018.
22. Raphael Leber. Ros openpose gossip, https://github.com/m0rph03nix/ ros openpose gossip. GIT, 2018.