Rosie ITL Agent
Rosie (RObotic Soar Instructable Entity) is an agent written in the
Soar Cognitive Architecture that learns new tasks from scratch through situated interactive instruction.
Rosie is part of research in the emerging area of Interactive Task Learning (ITL),
and has been used to learn novel games, puzzles, household tasks, perceptual features,
spatial relations, and hierarchical task concepts.
What sets Rosie apart is its ability to learn completely new tasks and concepts from a single example
and generalize its task knowledge in many different situations.
It does this in an online, interactive manner where it detects missing knowledge and performs
search or initiates new interactions to obtain that knowledge.
Rosie is a project at the University of Michigan Soar Lab, led by Professor John E. Laird.
- AAAI22 Demo
- Videos
- Publications
Contributors
- Mazin Assanie
- James Kirk
- John E. Laird
- Peter Lindes
- Aaron Mininger
- Shiwali Mohan
Learn More
Recent powerpoint presentation about Rosie
Archive of Learnable Puzzles and Games
Powerpoint of the history of ITL
Article in IEEE Magazine about ITL
AAAI22 Demo
AAAI22 Demo
We present a demonstration of the interactive task learning agent Rosie, where it learns the task of patrolling a simulated barracks environment through situated natural language instruction. In doing so, it builds a sizable task hierarchy composed of both innate and learned tasks, tasks formulated as achieving a goal or following a procedure, tasks with conditional branches and loops, and involving communicative and mental actions. Rosie is implemented in the Soar cognitive architecture, and represents tasks using a declarative task network which it compiles into procedural rules through chunking. This is key to allowing it to learn from a single training episode and generalize quickly.
Videos
Interior Guard
This video from showcases Rosie learning to perform an interior guard task in a simulated barracks environment. Highlights include learning ‘whenever’ tasks, composite tasks, looping tasks, tasks with conditional actions, and ability to interrupt an already learned task to extend behavior.
5-Puzzle with the Fetch robot
This video shows Rosie on the Fetch robotics platform solving a game of 5-puzzle after it had been taught through instruction. This is a variation of the 15-puzzle, where you must slide 15 scrambled tiles around to their original places.
Kitchen Tasks
In this video featuring the ai2thor simulator, Rosie learns where to store objects, how to heat something in the microwave, and how to serve water. Highlighted capabilities include learning conditional goals, learning actions that depend on user preferences, and incorporating temporal conditions into the task.
Learning Fetch
In this video featuring the Magicbot, Rosie learns to fetch an object from another location and bring it back. A key step in this task is that Rosie learns to remember the starting location so it can go back there later, thus generalizing to different starting places.
Learning Deliver
This video showcases Rosie learning the deliver task with the magicbot platform, where the goal is to deliver an object to the specified location. In the video Rosie also learns to associate a new name with a location.
Tower of Hanoi
Rosie learns the classic Tower of Hanoi puzzle with the tabletop robotic arm.
8 Puzzle
Rosie learns to solve the 8 puzzle (a variant of the 15 tile sliding puzzle) in a simulation of the tabletop robot arm.
Frogs and Toads
Rosie learns the Frogs and Toads jumping puzzle game. The video highlights the structures that are built up to recognize the actions, goal, and concepts such as ‘placed’.
Publications
Task Learning
Learning Hierarchical Symbolic Representations to Support Interactive Task Learning and Knowledge Transfer
2019 | James Kirk, John E. Laird
This paper explains Rosie’s approach to learning hierarchical task concepts used especially in learning game and puzzle tasks, and demonstrates the ability of the agent to transfer these concepts to new tasks and resolve ambiguities that arise.
Interactively Learning a Blend of Goal and Procedural Tasks
2018 | Aaron Mininger, John E. Laird
Explains how the task-learning part of Rosie was extended to learn simple procedural tasks in an integrated manner.
Learning Task Goals Interactively with Visual Demonstrations
2016 | James Kirk, Aaron Mininger, John E. Laird
Describes work that allows Rosie to learn task goals through visual demonstrations instead of linguistic descriptions. This was done with games, puzzles, and regular tasks on the tabletop robot.
Learning Goal-Oriented Hierarchical Tasks from Situated Interactive Instruction
2014 | Shiwali Mohan, John E. Laird
This paper details the first version of interactive task learning done in Rosie: learning goal-oriented hierarchical tasks in the tabletop domain using explanation based learning.
Learning Grounded Language through Situated Interactive Instruction
2012 | Shiwali Mohan, Aaron Mininger, James Kirk, John E. Laird
Describes an early version of the Rosie project with the tabletop blocks-world domain that can learn nouns (shapes), adjectives (colors), prepositions (spatial relations), and verbs (goal-based actions).
Language Comprehension and Interaction
Grounding Language for Interactive Task Learning
2017 | Peter Lindes, Aaron Mininger, John E. Laird
Describes the Lucia comprehension system which uses Embodied Construction Grammar approach to perform grounded language processing in Soar which is used by Rosie.
Towards an Indexical Model of Situated Language Comprehension
2014 | Shiwali Mohan, Aaron Mininger, John E. Laird
Describes a computational model of situated language comprehension that incorporates knowledge from perception, domain knowledge, and short and long term experiences when generating semantic representations.
Perceptual Reasoning
Using Domain Knowledge to Correct Anchoring Errors in a Cognitive Architecture
2019 | Aaron Mininger, John E. Laird
Details how Rosie uses its spatial memory to reason over perception and construct a model of the world that is robust to certain anchoring errors and perceptual noise.
Interactively Learning Strategies for Handling References to Unseen or Unknown Objects
2016 | Aaron Mininger, John E. Laird
Describes work done to extend Rosie to a mobile environment and how it handles references to objects that are not visible. Rosie learns different strategies for finding objects through instruction.
Other Rosie Work
Expanding Task Diversity in Explanation-Based Interactive Task Learning
2019 | Aaron Mininger - Thesis Proposal
This is an unpublished thesis proposal, but it gives a high-level overview of the current state of task learning in Rosie and some of the new capabilities that have been added.
From Verbs to Tasks: An Integrated Account of Learning Tasks from Situated Interactive Instruction
2015 | Shiwali Mohan - Thesis
Gives a very detailed account of Rosie, with a focus on interaction management and learning tasks. Although it is a bit outdated, the parts about how Rosie does dialog management and the theories behind learning from SII are still very relavent.
Interactive Task Learning
Interactive Task Learning (article)
2017 | Laird, Gluck, Anderson, Forbus, Jenkins, Lebiere, Salvucci, Scheutz, Thomaz, Trafton, Wray
Interactive Task Learning: Humans, Robots, and Agents Acquiring New Tasks Through Natural Interactions
2019 | John E. Laird, Kevin Gluck