Language, Action and Perception (APL-ESSLLI)

Introductory Course in Language and Computation at ESSLLI 2019,
Week 2, 17:00 - 18:30, 12 - 16 August, University of Latvia, Riga, Latvia

Course description

The course gives a survey of theory and practical computational implementations of how natural language interacts with the physical world through action and perception. We will look at topics such as semantic theories and computational approaches to modelling natural language, action and perception (grounding), situated dialogue systems, integrated robotic systems, grounding of language in action and perception, generation and interpretation of scene descriptions from images and videos, spatial cognition, and others.

As the course studies how humans structure and interact with the physical world and express it in language, it bridges into the domains of cognitive science, computer vision, robotics and therefore more broadly belongs to the field of cognitive artificial intelligence. Typical applications of computational models of language, action, and perception are image search and retrieval on the web, navigation systems that provide more natural, human-like instructions, and personal robots and situated conversational agents that interact with us in our home environment through language.

More details


  • 2019-08-05 Looking for a PhD? Come and work with us! One or several positions in Computational Linguistics available with CLASP in Gothenburg with a deadline of 19 August. You can find more information here or talk to us.
  • 2019-08-05 Welcome to the course! It will take place in Room 14 17:00 - 18:30 Monday to Friday of Week 2, 12 to 16 August.


The course gives a survey of approaches in linguistics, psychology, computer science and robotics related to computational interpretation and generation of language that is grounded in action and perception and takes into account geometric, cognitive, functional, or embodiment criteria in modelling. Topics that are included are:

  • Day 1: Introduction: language, action and perception
    Simon Dobnik, slides
  • Day 2: Language and space
    John Kelleher, slides; Mehdi Ghanimifard, tutorial; Simon Dobnik, experiment
  • Day 3: Generating and interpreting grounded language
    Simon Dobnik, slides; Mehdi Ghanimifard, tutorial
  • Day 4: Referring to what matters (attention)
    John Kelleher, slides; Mehdi Ghanimifard, tutorial
  • Day 5: Learning language with robots
    Simon Dobnik slides; Mehdi Ghanimifard, tutorial


Simon Dobnik
University of Gothenburg

Simon Dobnik

Simon Dobnik is an Associate Professor of Computational Linguistics and a member of the Centre for Language Technology (CLT) and the Centre for Linguistic Theory and Studies in Probability (CLASP), both at the University of Gothenburg. His research interests include spatial cognition, computational models of language and perception, human-robot interaction, situated spoken dialogue systems, and computational representations of meaning (semantics).

John D. Kelleher
University of Gothenburg

John D. Kelleher

John D. Kelleher is a Professor in Computer Science at the Technological University Dublin (TU Dublin), where he is the Academic Leader of the ICE research institute. John is also the head of the ADAPT Research Centre at TU Dublin which is funded by the Science Foundation Ireland grant (RC/13/2106). John has published a number of books with MIT Press including Deep Learning (Kelleher, 2019), Data Science (Kelleher and Tierney, 2018), and a textbook on machine learning (Kelleher, Mac Namee, and D'Arcy, 2015). John’s interests in the area of spatial cognition relate to developing cognitively inspired models that enable computational systems (such as robots) to ground spatial language in sensor data. In particular, John is interested in the role of perceptual phenomenon (such as visual attention, object occlusions and viewer perspective) on the semantics of spatial terms. John is also interested in situated dialog and the interaction between models of working memory, attention and reference.

Mehdi Ghanimifard
University of Gothenburg

Mehdi Ghanimifard

Mehdi Ghanimifard is a PhD candidate in Computational Linguistics. His research area is grounded language understanding and generation with neural language models. He is interested in examining models which combine linguistic representations and uncertain perceptual representations in a single framework.