SpiceBOT: Robotic Handling of Spice Sacks

Short Facts

  • Generative AI pipeline for generating and automatically labeling data.
  • Photo-realistic simulation of robot environment using Isaac Sim
  • Partner: Kräuter Mix GmbH

Project Team

Project Description

This project’s goal is to develop a robotic handling solution for deformable spice sacks that can weigh up to 30kg. The system will be deployed for depalletizing incoming sacks at our partner company Kräuter Mix GmbH.

We address two main challenges: 1) manipulating deformable sack-like objects, and 2) training an accurate, robust model to segment and localize the sacks despite limited available data.

To overcome the data scarcity, we developed two complementary synthetic data generation pipelines. The first relies on photorealistic simulation using BlenderProc while the second leverages Generative AI (GenAI) through Stable Diffusion and other foundation models for automated labeling. 

Using a text-to-dataset approach, the GenAI pipeline produces fully labeled datasets directly from natural-language prompts. Our results show that combining GenAI-generated data with photorealistic simulation data yields a high-quality training set, enabling effective segmentation and grasp-pose detection that transfers reliably to real-world scenarios. 

 Data Generation Pipeline using GenAI

Data Generation Pipeline using BlenderProc

Segmentation Plastic Sacks

Segmentation Paper Sacks