End-To-End and Direct Human-Flying Robot Interaction

As the application domain of Unmanned Aerial Vehicles (UAV) expands to the consumer market and with recent advances in robot autonomy and ubiquitous computing, a new paradigm for human-UAV interaction has started to form. In this new paradigm, humans and UAV(s) are co-located (situated) and use natural and embodied interfaces to share au- tonomy and communicate. This is in contrast to the traditional paradigm in Human-UAV interaction in which the focus is on designing control interfaces for remotely operated UAVs and sharing autonomy among Human-UAV teams. Motivated by application domains such as wilderness search and rescue and personal filming, we define the required components of end-to-end interaction between a human and a flying robot as interaction initiation (ii) approach and re-positioning to facilitate the interaction and (iii) communication of intent and commands from the human to the UAV and vice versa. In this thesis we introduce the components we designed for creating an end-to-end Human-Flying Robot Interaction sys- tem. Mainly (i) a fast monocular computer vision pipeline for localizing stationary periodic motions in the field of view of a moving camera; (ii) a cascade approach controller that combines appearance based tracking and visual servo control to approach a human using a forward-facing monocular camera; (iii) a close-range gaze and gesture based interaction system for communication of commands from a human to multiple flying UAVs using their on-board monocular camera; and (iv) a light-based feedback system for continuous commu- nication of intents from a flying robot to its interaction partner. We provide experimental results for the performance of each individual component as well as the final integrated sys- tem in real-world Human-UAV Interaction tests. Our interaction system, which integrates all these components, is the first realized end-to-end Human-Flying Robot Interaction sys- tem whereby an uninstrumented user can attract the attention of a distant (20 to 30m) autonomous outdoor flying robot. Once interaction is initiated, the robot approaches the user to close range (≈ 2m), hovers facing the user, then responds appropriately to a small vocabulary of hand gestures, while constantly communicating its states to the user through its embodied feedback system. All the software produced for this thesis is Open Source.