View on GitHub


Soar Agent (and SML code) that learns through situated interactive instruction in a robotic environment

Rosie ITL Agent

Rosie (RObotic Soar Instructable Entity) is an agent written in the Soar Cognitive Architecture that learns new tasks from scratch through situated interactive instruction. Rosie is part of research in the emerging area of Interactive Task Learning (ITL), and has been used to learn novel games, puzzles, household tasks, perceptual features, spatial relations, and hierarchical task concepts.
What sets Rosie apart is its ability to learn completely new tasks and concepts from a single example and generalize its task knowledge in many different situations. It does this in an online, interactive manner where it detects missing knowledge and performs search or initiates new interactions to obtain that knowledge. Rosie is a project at the University of Michigan Soar Lab, led by Professor John E. Laird.

- AAAI22 Demo
- Videos
- Publications


Learn More

Recent powerpoint presentation about Rosie

Archive of Learnable Puzzles and Games

Youtube Channel

Soar Lab at Michigan

Powerpoint of the history of ITL

Article in IEEE Magazine about ITL

Recent book describing ITL

AAAI22 Demo

AAAI22 Demo

AAAI22 Demo

We present a demonstration of the interactive task learning agent Rosie, where it learns the task of patrolling a simulated barracks environment through situated natural language instruction. In doing so, it builds a sizable task hierarchy composed of both innate and learned tasks, tasks formulated as achieving a goal or following a procedure, tasks with conditional branches and loops, and involving communicative and mental actions. Rosie is implemented in the Soar cognitive architecture, and represents tasks using a declarative task network which it compiles into procedural rules through chunking. This is key to allowing it to learn from a single training episode and generalize quickly.

AAAI22 Demo poster

AAAI22 Paper


Interior Guard

Interior Guard

This video from showcases Rosie learning to perform an interior guard task in a simulated barracks environment. Highlights include learning ‘whenever’ tasks, composite tasks, looping tasks, tasks with conditional actions, and ability to interrupt an already learned task to extend behavior.

5-Puzzle with the Fetch robot

Fetch Robot

This video shows Rosie on the Fetch robotics platform solving a game of 5-puzzle after it had been taught through instruction. This is a variation of the 15-puzzle, where you must slide 15 scrambled tiles around to their original places.

Kitchen Tasks

Kitchen Tasks

In this video featuring the ai2thor simulator, Rosie learns where to store objects, how to heat something in the microwave, and how to serve water. Highlighted capabilities include learning conditional goals, learning actions that depend on user preferences, and incorporating temporal conditions into the task.

Learning Fetch

Fetch Task

In this video featuring the Magicbot, Rosie learns to fetch an object from another location and bring it back. A key step in this task is that Rosie learns to remember the starting location so it can go back there later, thus generalizing to different starting places.

Learning Deliver

Deliver Task

This video showcases Rosie learning the deliver task with the magicbot platform, where the goal is to deliver an object to the specified location. In the video Rosie also learns to associate a new name with a location.

Tower of Hanoi

Tower of Hanoi

Rosie learns the classic Tower of Hanoi puzzle with the tabletop robotic arm.

8 Puzzle

8 Puzzle

Rosie learns to solve the 8 puzzle (a variant of the 15 tile sliding puzzle) in a simulation of the tabletop robot arm.

Frogs and Toads

Frogs And Toads

Rosie learns the Frogs and Toads jumping puzzle game. The video highlights the structures that are built up to recognize the actions, goal, and concepts such as ‘placed’.


Task Learning

Learning Hierarchical Symbolic Representations to Support Interactive Task Learning and Knowledge Transfer
2019 | James Kirk, John E. Laird

This paper explains Rosie’s approach to learning hierarchical task concepts used especially in learning game and puzzle tasks, and demonstrates the ability of the agent to transfer these concepts to new tasks and resolve ambiguities that arise.

Interactively Learning a Blend of Goal and Procedural Tasks
2018 | Aaron Mininger, John E. Laird

Explains how the task-learning part of Rosie was extended to learn simple procedural tasks in an integrated manner.

Learning Task Goals Interactively with Visual Demonstrations
2016 | James Kirk, Aaron Mininger, John E. Laird

Describes work that allows Rosie to learn task goals through visual demonstrations instead of linguistic descriptions. This was done with games, puzzles, and regular tasks on the tabletop robot.

Learning Goal-Oriented Hierarchical Tasks from Situated Interactive Instruction
2014 | Shiwali Mohan, John E. Laird

This paper details the first version of interactive task learning done in Rosie: learning goal-oriented hierarchical tasks in the tabletop domain using explanation based learning.

Learning Grounded Language through Situated Interactive Instruction
2012 | Shiwali Mohan, Aaron Mininger, James Kirk, John E. Laird

Describes an early version of the Rosie project with the tabletop blocks-world domain that can learn nouns (shapes), adjectives (colors), prepositions (spatial relations), and verbs (goal-based actions).

Language Comprehension and Interaction

Grounding Language for Interactive Task Learning
2017 | Peter Lindes, Aaron Mininger, John E. Laird

Describes the Lucia comprehension system which uses Embodied Construction Grammar approach to perform grounded language processing in Soar which is used by Rosie.

Towards an Indexical Model of Situated Language Comprehension
2014 | Shiwali Mohan, Aaron Mininger, John E. Laird

Describes a computational model of situated language comprehension that incorporates knowledge from perception, domain knowledge, and short and long term experiences when generating semantic representations.

Perceptual Reasoning

Using Domain Knowledge to Correct Anchoring Errors in a Cognitive Architecture
2019 | Aaron Mininger, John E. Laird

Details how Rosie uses its spatial memory to reason over perception and construct a model of the world that is robust to certain anchoring errors and perceptual noise.

Interactively Learning Strategies for Handling References to Unseen or Unknown Objects
2016 | Aaron Mininger, John E. Laird

Describes work done to extend Rosie to a mobile environment and how it handles references to objects that are not visible. Rosie learns different strategies for finding objects through instruction.

Other Rosie Work

Expanding Task Diversity in Explanation-Based Interactive Task Learning
2019 | Aaron Mininger - Thesis Proposal

This is an unpublished thesis proposal, but it gives a high-level overview of the current state of task learning in Rosie and some of the new capabilities that have been added.

From Verbs to Tasks: An Integrated Account of Learning Tasks from Situated Interactive Instruction
2015 | Shiwali Mohan - Thesis

Gives a very detailed account of Rosie, with a focus on interaction management and learning tasks. Although it is a bit outdated, the parts about how Rosie does dialog management and the theories behind learning from SII are still very relavent.

Interactive Task Learning

Interactive Task Learning (article)
2017 | Laird, Gluck, Anderson, Forbus, Jenkins, Lebiere, Salvucci, Scheutz, Thomaz, Trafton, Wray

Interactive Task Learning: Humans, Robots, and Agents Acquiring New Tasks Through Natural Interactions
2019 | John E. Laird, Kevin Gluck