In the second period, the project will proceed from static scenes to dynamic scenarios and focus on how local environmental changes determine global structural shifts of the overall situation model incorporating intentionality and person activity. The question addressed of how shared situation models arise will be extended to environmental changes, protagonist activities, and adaptations over time, focussing communication about a dynamically changing scene. It will investigate dynamic conceptual transitions and how local environmental changes determine global structural shifts of the overall situation model. In the  envisioned scenarios, one interlocutor rearranges furniture, decorations and other items in a room and, thereby, introduces functional changes of specific room areas and global changes of the conceptual structure of the room. This activity is either instructed or actively observed and commented on by the second interlocutor. We assume that such structural changes of the mental model are driven by automatic alignment processes that lead to an anticipation of the final global scene configurations before it is actually perceived. This leads to an active perception strategy of the interlocutors in that they pro-actively change their physical viewpoint of a scene, change their vocabulary for referring to that scene, change the use of possible anchor objects, reference frames, levels of granularity, and the attribution of saliency to room components. The project will explore what kinds of factors determine and influence changes from one room category to another and functional changes of specific room areas.

Language processing involves the formation of a representation of the matters talked about – besides a representation of the language itself. For a dialogue to be successful, it is essential that both interlocutors develop partially aligned models of the situation under discussion. We assume that this holds also for communication about complex, visually perceived environments and propose that they are mentally represented by selective situation models based on given spatial descriptions and 3D perception.

The main goal of the project is to investigate the process of building a shared perceptually grounded mental model of the situation under discussion through conversation and to exploit the principles of human-human situation-model aligning for human-robot communication. This will enable a mobile robot to more effectively understand descriptive language, to deal with partially unknown situations, to ignore irrelevant information and to act in cluttered environments. We intend to examine factors that lead to an alignment of such representations in situational models. We focus on relational and perceptual components of such situation models that are grounded in a common visually perceived environment. Several aspects of the question how shared situation models arise will be addressed in the project by combining psycholinguistic experiments with theoretical and conceptual work in cognitive science, and computational modelling. These include the interplay between automatic alignment processes and resource-limited elaborate construction, the essentially resource-free use of these integrated situation models once they are formed, and the role of pan-situational world knowledge (schemata, categorisation levels, typicality) invoked by language use.

>