RetrDex: Efficient Object Retrieval in Cluttered Scenes with a Dexterous Hand

Supplementary

RetrDex a framework that learns policies for efficient object retrieval in simulation and transfers zero-shot to real-world deployment.

Abstract

Retrieving objects buried beneath clutter is both challenging and time-consuming, as complex support relationships make manipulation particularly difficult. Existing methods either focus on support relations and rely on sequential grasping to remove occluding objects, which is inefficient and assumes feasible grasps for every occluder, or perform preparatory actions such as pushing to facilitate subsequent grasps, still treating physical interactions as isolated auxiliary steps. In this paper, we propose RetrDex, an efficient framework for dexterous arm–hand systems to learn object retrieval in cluttered scenes. Our approach leverages large-scale parallel reinforcement learning in diverse, carefully designed cluttered scenes and incorporates a spatially aware representation that encodes occlusion patterns and spatial relationships among the target, the dexterous hand, and surrounding clutter. This structured representation enables the policy to develop emergent manipulation skills (e.g., pushing, stirring, and poking) that actively clear occluders. We evaluate RetrDex on 16 household objects across varied clutter configurations, achieving superior retrieval performance and efficiency on both seen and unseen targets. Furthermore, we demonstrate successful zero-shot transfer to a real-world dexterous multi-fingered robot system, validating the practical applicability of our method.

Method

Method overview
Overview of the RetrDex. (a) A graph neural network (GNN) module learns a spatial-aware representation that encodes occlusion and spatial relationships among objects. (b) Using this representation, a teacher policy is trained in simulation to acquire effective retrieval strategies. (c) High-quality trajectories generated by the teacher are filtered and used to train a student policy via imitation learning, enabling efficient deployment on a real robot.

RL Policy in Simulation

Yellow box shows the target object in the training set, while blue box shows the target object in the test set

Target Object Position Generalization

The target objects are placed in distinct regions of the box: top-left, bottom-left, top-right, and bottom-right.

Retrieval Efficiency

Conclusion

We presented RetrDex, a framework for efficient object retrieval in cluttered scenes with dexterous multi-finger hands. Unlike methods that rely on sequential removal or treat pre-grasp interactions as auxiliary steps, our approach learns a continuous spectrum of interaction skills—such as pushing, stirring, and poking—that actively expose target objects. By combining large-scale RL training in simulation with a spatial-aware representation of occlusion and contact relations, RetrDex achieves efficient and robust manipulation in high-dimensional, contact-rich settings. Experiments in both simulation and the real world confirm superior retrieval performance, generalization to diverse objects, and zero-shot transfer to physical robots. Future directions include integrating advanced perception and reasoning modules to move toward fully autonomous active exploration in unstructured environments.