Vision-Based Solver

Shoebox-tasks have a clear visual structure, making them well suited for automated vision-based analysis. Our system implements a perception–analysis–action loop using a Vision–Language Model (VLM).

Starting from an image of a Shoebox-task setup and a general PDDL domain, it interprets the visual scene and generates a corresponding PDDL problem instance and action plan. After each action execution, a new image is analyzed to detect changes, revise the PDDL state and update the plan. This loop continues until the task is completed or an invalid state is detected.

The robot can either solve the task autonomously or support a student by guiding task execution through demonstrations, gestures, voice instructions, and real-time error monitoring.