SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting

Introduction

The Sim2Real transfer problem in robotics occurs when robots trained in simulated environments fail to perform effectively in the real world. This challenge is particularly prominent in RGB-based manipulation tasks, where the visual differences between simulated and real-world data lead to poor transfer performance. Addressing this gap is critical as simulations provide a scalable, cost-effective training solution, while real-world training is often expensive and time-consuming. SplatSim introduces a novel framework leveraging Gaussian Splatting to bridge the Sim2Real gap, enabling robots to learn manipulation tasks in photorealistic simulated environments. The trained policies can then be deployed directly in real-world scenarios without additional fine-tuning.

Objective

To develop a framework that:

Utilizes Gaussian Splatting to enhance the photorealism of simulated environments.
Facilitates zero-shot transfer of manipulation policies from simulation to real-world tasks.
Reduces reliance on real-world data collection by providing an efficient, automated simulation-based training pipeline.

Methodology

Gaussian Splatting: The framework replaces traditional 3D object representations (e.g., polygon meshes) with Gaussian Splats, simulating realistic light interactions and enhancing visual fidelity.
Physics Integration: It integrates physics engines to model interactions like collisions and gravity, creating a realistic training environment.
Automated Data Generation: Demonstrations are generated in the simulator using motion planners or teleoperation, significantly reducing human effort.
Behavior Cloning: Policies are trained using high-quality synthetic data generated by the simulator.
Evaluation: Policies trained in the simulator are deployed in the real world across various tasks without additional fine-tuning.

Results

SplatSim was evaluated on four real-world manipulation tasks:

T-Push: 90% success rate
Pick-Up-Apple: 95% success rate
Orange-On-Plate: 90% success rate
Assembly: 70% success rate

The average success rate of 86.25% for zero-shot Sim2Real transfer closely approaches the 97.5% success rate of policies trained on real-world data. Furthermore, data collection in simulation required only 3 hours compared to 20.5 hours for real-world data collection, showcasing SplatSim's efficiency and scalability.

Conclusion

SplatSim demonstrates that combining photorealistic rendering techniques with robust simulation-based training can significantly narrow the Sim2Real gap in RGB-based robotic manipulation tasks. By enabling zero-shot transfer and reducing the need for real-world data, SplatSim establishes a scalable and efficient framework for robotic training. Future work aims to extend the framework to more complex tasks involving deformable objects, potentially benefiting fields such as industrial automation and precision agriculture.

Contribution

I was responsible for the early stage design and implementation of the Gaussian Splatting module in the SplatSim framework. This involved developing gaussian robot segmentation pipeline, grounding the gaussian robot to the simulator robot, and transferring the robot trajectories from the sim to the gaussian space. I also wrote the data curation scripts for the diffusion policy and collected real world demonstrations for the baseline evaluation. Apart from this, I also contributed to the paper writing by creating vectorized figures. This research was conducted at the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA.