Hinomiyagura: Proposal of robot new task using more human sensors in real environments

Tomoichi Takahashi, Masaru Shimizu

Meijo University; Chukyo University

Abstract In recent times, during natural disasters, robots have played an important role in search and rescue operations in places that are not easily accessible to humans. Mobility in rough terrain, monitoring of the surroundings to search for victims, and creating disaster maps are key functions of robots in search and rescue operations. Hence, simulations should provide more realistic information more naturally. This is made possible using sound. This TDP proposes and discusses the need for simulating inspection tasks with sound information. Proposals of new tasks using sound are presented. A prototype shows that the usage of sound makes the applications of robot simulation more robust.

Keywords: Simulation with sound field, Rescue simulation, Infrastructure maintenance, Human in loop

1 Introduction

The RoboCup Rescue League aims to mitigate the losses caused by disasters and emergencies by supporting the development of robots. On September 11, 2001, many robotics researchers brought their robots to search the wreckage of the World Trade Center[8]. The robots surveyed and reported the status of the fields where unforgiving environments prevented the activities of rescue workers in the area. Human rescuers uses their own sensors to get ambient information not only viewing and sound, but also thermo, smell and so on. A rescue league should have tasks including not only passive image recognition and passive sound recognition but also active action evaluation like hammering test to evaluate performances of rescue robots used instead of human. A decade later, a robot that participated in RoboCup was used to investigate inside the buildings of the Fukushima Daiichi Nuclear Power (FDNP). Robots were also used in difficult conditions during the initial stages of the FDNP accident [15] [6]. After this initial stage, the FDNP decommissioning plan was announced and new robots were designed and developed for use for this plan over the next several decades [13].

Aging infrastructure, such as bridges and tunnels, exert severe pressure on human society. After several decades in service, the robustness of the infrastructure decrease. Maintenance and replacement are the common measures taken to enhance the stability of such aging infrastructure. Robot technology is believed to be effective for both inspection and maintenance of infrastructure as well as rescue operations [1]. Setting standard tasks and contesting robots' performance accelerate research and development of robots in the application elds.

In the RoboCup Rescue Virtual Robot League (RVRL), rescue tasks have been modeled to verify algorithms installed in and the operations of rescue robots [10]. For example, map generation and victim searches have been set as typical tasks. In reality, these tasks are performed in challenging environments: the floors are filled with debris, it is dark, the air is contaminated with dust, sounds of noise and diffracted and fluctuated Wi-Fi among other challenges [12][11]. Simulation platforms that represent robot activities in such environments provide realistic test fields to meet not only the needs of the rescue operation but also those of the inspection tasks at plants, bridges, tunnels, and other locations [4].

In this paper, we discuss tasks where sound plays an information role and propose a simulation platform that incorporates the sound field. Rescue robot contests and their simulation platforms are surveyed in Section 2. In Section 3, robots' activities with sound information are discussed and new standard tasks are proposed. A sample task is demonstrated in Section 4. In Section 5, a discussion and summary of the paper are presented.

Table 1. History of Rescue Robot Competition and Test Field

2 Rescue Robot Contests and Simulation Platforms

During disasters, human rescuers use all their sensing abilities: sight, hearing, smell, and touch to explore the fields, to search for victims, and to assure their own safety. With regard to infrastructure maintenance, nondestructive testing (NDT) plays a key role in validating the tunnel structures, bridge components, and pipe connections in plants. Some NDT inspection tasks have been completed by workers with suitable devices, such as a hammer. Fig.1 shows visual inspection being carried out and a hammering test.

Robots are designed to perform tasks in specific applications. The necessary sensors are mounted. Many projects on rescue tasks and robots are promoted in the form of robotics competitions. Their purposes range from search-and-rescue operations at disaster sites to inspections of social infrastructure and oil platforms. In some competitions, simulation platforms are provided to develop and assess the robots in concert with the tasks in real competitions (Table 1). The RoboCup RVRL, the Gazebo used in the DARPA Robotics Challenge (DRC) [3][5], and the Japan Virtual Robotics Challenge[7] are such examples.

Almost all simulated fields for evaluating rescue robot performance are set to reproduce real-life situations that are composed of a lot of elements: complex terrain, rubble, moving objects, diffracting light, sounds from the environment, temperature, humidity, CO2, wind, gas, unstable wireless condition, etc. At present, the situation fields are simple, such as flat floors, uniform lighting, no sound, etc. Table 2 lists typical scenarios that appear in tough environments; this table indicates that sound is one of the key factors in recognizing the situation around a robot.

Table 2. Scenarios, sounds and robot actions

3.1 Scenarios with sound information

In RVRL, sound information will give some evaluation scenes in game missions. Robots will run in various scenarios like real emergencies. Fig. 2 shows one scenario where a robot will be used in case of emergencies. The robot enters the disaster site to explore the site and the primary task of RoboCup RVRL is to map the inside of the site. However, while performing the mission, the robot finds an injured person who needs help and the robot operator corresponds with the person through the robot. On moving deeper inside the site, the operators of the robots may notice the sounds of water and gas leaks or unstable state of walls and stairs that are caving in. Those sound information plotted in a search result map should be counted as game points.

Our proposal is that the simulation platform should represent such tasks. The illustrated scenario is composed of the following three tasks.

T1 - Search and monitor the injured

Robot operator notices an injured person through cameras or microphones, approaches the person using the robot, and asks their life status through speakers and microphones installed on the robot. When they respond, the operator changes the current mission into a new one. The points to note are as follows:

• Through the robot's speaker and microphones, the person and the operator communicate with each other about the conditions of the person and the surrounding environment.
• When the person provides new information on the task, for example, that there are more people in the next room or that the nearby area is completely corrupted, the information improves the rescue performance.

T2 - Inspection gas and water leakage task

Laboratory buildings and factories are generally equipped with gas and water pipelines, and the fall of furniture can damage these pipelines. Checking for gas leaks prevents potential accidents. The leaking sounds of gas and water are important for the operator to notice the leaks.

T3 - Checking inside area

The damaged floors, doors, and stairs of inside the buildings will prevent robots from moving deeper inside and a collapse may damage the robots. The sounds generated by the robot's movements and NDT are a key indicator of the possibility of collapse.

3.2 Sound caused interactions with environment

Sound is generated while performing the aforementioned three tasks, and it consists of information that plays an important role in executing the tasks. Followings are ways in which sound is generated in those tasks:

• Sound 1: Communication between the robot operator and the injured person is facilitated by making the injured person (object) reply orally while the robot (operator) speaks to the object through speakers.
• Sound 2: The leakage of water or gas constantly generates sound. When the operators notice the sound through microphones, they direct the robot hand to shut the valves. The operations are confirmed by the changes in sound.
• Sound 3: Additionally, sound is generated by interaction between objects. Test of palpation, - percussion and auscultation - using devices that are used to diagnose the status of the targets and the sounds created by moving on unstable floors are used to proceed further or not. These sounds are created by collision between objects of robots and environments.

4 Proposal to add new task for RVRL

Figure 3 shows a task that we propose for the virtual robot league. The mission of the robot is to map the inside of a room and inspect one wall by the hammering test. The robot in the demonstration is a centaur type robot[2]. This type of robot won the first prize at JVRC that was held at 2015 and the hammering test was set as one of tasks. The robot has one microphone and one camera with which the operator can see ahead, but not the robot's footsteps simultaneously.

Figure 3 (b) is the top view of the room that the robot explores and the generated map by SLAM. The right wall is assigned to be inspected by the hammering test. The lower half part of the wall is vertical and the upper half part is inclined with 45°. The test points are assigned on the wall for inspection in the game. For example, the + signs in Figure 3 (c) are the points to be tested.

The operator is asked to inspect the wall by testing equally spaced 12 points. Screenshots in Figure 3 (d) show how he control the robot to hammer the wall and Algorithm 1 shows the flow of robot operations. The function ROBOT-HAND OPERATION is repeated during the inspection task and the INTERACTION-SOUND function is called at every human operation so that the operator can determine the effectiveness of the hammering operations by hearing the generated sound.

The operators hear the sound through the microphone, and they are expected to judge whether the hammering strikes the wall properly or not. The operations are valued with the consistency between the conditions used in generating sound and the results of their decisions. For example, the operator is expected to repeat the hammering test at a point where quality of the sound is not proper. At another point where the hammering sound is generated properly, the monitored sound is sent to the operator and are used for diagnosis of walls[14]. And it is expected for the robot task to move and test other points. The action sequences of operations are scored and the points are added to the evaluation points of executing tasks.

Algorithm 1 robot operations in rescues

procedure robot-hand operation
   approach assigned point
   manipulate hand to hammer the point
   check the hammer sound
   if the sound is abnormal then
      repeat hammering test
   end if return
end procedure

5 Discussion and Summary

Tasks of RoboCup RVRL was designed from the points that the rescue robots are expected to operate as first responders in the event of a disaster. It has been recognized that the functions required in rescue robots are similar in many respects to those required of service robots employed in daily infrastructure maintenance tasks and we can evaluate rescue robots with those tasks like ARGOS in RVRL. And, in developing robot systems that human is in-loop, it is thought to be useful to simulate the robot motions and the interaction with human operations simultaneously as they are.

In this paper, we propose a realistic simulation platform using sound and discuss the importance of the information in sound in performing operations and maintenance tasks of infrastructures. Our prototype takes a hammering test as an example of task of RoboCup RVRL. In the task, the robot operator decides the next course of action based on the sound information. The prototype demonstrates that the usage of sound makes the applications of robot simulation more robust.

As the service images in Fig. 2 indicates that the basic functions are common other field simulations, such as @Home leagues. Our proposal of a simulation platform with a sound field is not limited to tasks of rescue and maintenance; our system can be applied to other fields.

References

1. http://www.rita.dot.gov/publications/technology scan/robots, date:10.June. 2016.
2. T. Darmstadt. centaur robot tutorial. https://github.com/tu-darmstadt-rospkg/centaur robot tutorial, date:31. Mar. 2017.
3. DARPA Robotics Challenge. http://www.theroboticschallenge.org/. date:22. Feb. 2016.
4. Decommision Plan of Fukushima Daiichi Nuclear Power. http://www.tepco.co.jp/en/decommision/planaction/index-e.html, date:31. Mar. 2017.
5. Gazebo. http://gazebosim.org/. date:6. April. 2017.
6. E. GUIZZO. Fukushima robot operator writes tell-all blog. http://spectrum.ieee.org/automaton/robotics/industrial-robots/fukushimarobot-operator-diaries, date:21 Feb. 2015.
7. Japan Virtual Robotics Challenge. http://www.jvrc.org/en/index.html. date:22. Feb. 2016.
8. R. R. Murphy. Disaster Robotics. The MIT Press, 2014.
9. OpenAL. www.openal.org/, date:21. June. 2016.
10. RoboCup Virtual Robot League. http://wiki.robocup.org/Rescue Simulation League, date:31. Mar. 2017.
11. M. SHIMIZU and T. TAKAHASHI. Databased fluctuating wi-fi signal simulation environment for evaluating the control of robots. Journal of Japan Society for Fuzzy Theory and Intelligent Informatics, 29(2):567–573, 2017.
12. M. Shimizu and T. Takahashi. Simulation platform for performance test for robots and human operations. AAAI Fall Symposium Series Technical Reports FS-11-05, Nov:61–66, 2011.
13. TEPCO. http://www.tepco.co.jp/en/decommision/index-e.html, date:22. Feb. 2016.
14. A. Watanabe, J. Even, L. Y. Morales, and C. T. Ishi. Robot-assisted hammer sounding inspection of infrastructures. Journal of the Robotics Society of Japan, 33(7):548–554, 2015.
15. T. Yoshida, K. Nagatani, S. Tadokoro, T. Nishimura, and E. Koyanagi. Improvements to the rescue robot quince toward future indoor surveillance missions in the fukushima daiichi nuclear power plant. http://www.astro.mech.tohoku.ac.jp/ keiji/papers/pdf/2012-FSR-Yossyonline.pdf, date:30. Mar. 2017.