This research aims to explore the features of user interaction with georeferenced information by using the physical environment and its elements as interface. The use of geotags in digital files and the popularization of mobile computing established a new model of information-based interaction. It is up to the designer to create systems and interfaces that take into account the user location when accessing the surrounding information. It was identified that the screen-based solutions and other technological devices that constrain the relation between the user and the environment, interfere with the interactive experience. As an outcome of the research, a mobile application was developed that spares the use of displays at the act of selecting points of interest in the physical environment interaction. The system is based on smartphones' location and orientation sensors data to establish the interaction with the georeferenced and modeled surrounding elements. Thus, the user points the device and directly select the object of interest, in order to receive the data assigned to it. Without the interference of screens or immersive devices, the environment itself is presented as an interaction interface, avoiding the overlapping of new icons or symbols and making technology more subtle in its daily use.