Towards embodied agents for navigating and recognizing human activity

In this dissertation, we aim to make progress toward building embodied agents that coexist with humans and assist humans to perform various tasks physically. We focus on developing visual perception and navigational capabilities for embodied agents in human environments, and we explore three aspects of visual perception and navigation - human activity recognition, crowd navigation and agent morphology design. We investigate three research questions to approach the above three aspects of perception and navigation: (1) Can we improve an embodied agent's visual recognition of an ongoing human activity by utilizing its mobility? (2) Can we develop safe and efficient navigation behaviors of embodied agents by anticipating human future trajectories and modelling relations? (3) Can we efficiently design an agent's morphology automatically for optimal task performance? To address these research questions, we present three works that provide possible solutions. First, we introduce the embodied human activity recognition problem, where an agent moves in a 3D environment to classify ongoing human activities. We propose a reinforcement learning approach that learns a policy controlling the agent's movements over time, with the goal of acquiring new views that lead to accurate human activity classification. Second, we present a relational graph learning approach for robotic crowd navigation using model-based deep reinforcement learning that plans actions by looking into the future. Third, we present a continuous multi-fidelity Bayesian Optimization framework that efficiently utilizes computational resources via low-fidelity evaluations for automatic morphology design.