Virtual reality simulators have a serious flaw: Users tend to get lost and disoriented as they navigate. The prevailing opinion is that this is due to the lack of physical motion cues, but a growing body of research challenges this notion. In two experiments, 48 participants estimated their position after passive motions in a virtual environment without landmarks (ranging from pure optic flow to a structured city), by pointing towards the origin of the simulated movement. In half of the trials the visually displayed turns were accompanied by a matching physical rotation. Results showed that while physical rotation cues did not improve spatial orientation performance, structured visuals did. Furthermore, we observed that visuals experienced first by a participant significantly affected spatial orientation performance in subsequent environments. Our findings lend support to the notion that spatial orientation ability in VR may not require physical motion cues, but can be facilitated by a naturalistic and structured environment. This knowledge improves our understanding of how different modalities affect human spatial cognition, and can guide the design of safer and more affordable VR simulators.