Generalization Challenges for Real-World Deployment
Workshop Day: 14/October/2024 | Time: 9. am to 1 pm | Room 10
Best Poster Award
Abstract
As consumer needs and industry demands evolve, there is an increasing call for robust and adaptable mobile manipulation solutions for efficient and reliable robot deployment across varied environments, from factories to homes. Building general-purpose mobile manipulation systems that scale beyond specific scenarios and adapt to novel conditions remains a great challenge in robotics. Although considerable progress has been made in advancing mobile manipulation capabilities and skills, a notable gap persists in addressing the generalization necessary for confident deployment in real-world environments.
Robot learning facilitates autonomous skill acquisition, adaptation to unseen and dynamic environments, and execution of complex tasks. While learning-based approaches show success in controlled settings and specific tasks, achieving generalization in real-world deployment remains an open challenge.
This workshop explores existing and envisioned mobile manipulation systems, technologies, and learning methods with a focus on adaptability, robustness, and generalization in diverse real-world settings. Key questions include building an ideal mobile manipulation system, effective generalization under domain and deployment gaps, coordination of generalizable skills for long-horizon tasks, leveraging multi-modal sensory information, control architectures and end-to-end learning (vision-language-action models), addressing linguistic and interactive features for human-centric environments, utilizing simulated environments for real-world applications, managing uncertainties in sensing/actuation during deployment, strategies for continuous skill adaptation, and measuring generalization.
The workshop encourages poster submissions on innovative mobile manipulation generalization solutions capable of handling diverse real-world complexities and tasks. The workshop seeks to foster a research community, facilitating collaboration among researchers, industries, and entrepreneurs to advance mobile manipulation robots and learning solutions for practical deployment.
SCOPE
- Ideal Mobile Manipulation System: Designing mobile manipulators with optimal embodiments, on-board sensing, and end tools to support generalization in diverse scenarios.
- Skill Generalization and Safety: Developing methods for mobile manipulators to generalize learned skills, ensuring safety, adaptability, and human interaction in dynamic real-world environments.
- Simulation to Reality (Sim2Real): Utilizing simulated environments for effective robot training, focusing on skill transfer and generalization from simulation to real-world applications.
- Human-Centric Features: Identifying crucial linguistic, interactive, and safety features for mobile manipulators to perform complex tasks in human-centric environments.
- Control Architectures: Exploring how various control architectures impact the speed, reliability, generalization, and robustness of mobile manipulation.
- Managing Uncertainties and Continuous Adaptation: Addressing uncertainties in sensing and actuation, and developing strategies for continuous skill adaptation, ensuring robust and reliable performance in diverse real-world settings.
Speakers
Karlsruhe Institute of Technology, Germany
University of Illinois Urbana-Champaign
Tsinghua University
Carnegie Mellon University
Google DeepMind
Organizers
- Rajkumar Muthusamy ( Dubai Future Foundation)
- Paolo Dario ( Scuola Superiore Sant’Anna, Pisa)
- Tarek Taha ( Dubai Future Foundation)
- Georgia Chalvatzaki (TU Darmstadt)
- Kensuke Harada (Osaka University)
- Yuqian Jiang (UT Austin)
- Roberto Martin-Martin (UT Austin)
- Weiwei Wan (Osaka University)
Program Overview
| Time | Speaker and Topic | Duration | |
| Start | End | ||
| 09:00 | 09:05 | Dr. Rajkumar Muthusamy, Dubai Future Foundation Title: Introduction to the Workshop | 5’ |
| 09:05 | 09:35 | Prof. Guyue Zhou, Tsinghua University Title: Utilizing Simulation and Reality Data for Real-world Mobile Manipulation | 20‘ + 10‘(Q&A) |
| 09:35 | 10:05 | Prof. Deepak Pathak, Carnegie Mellon University Title: Pretraining and Adaptation for Scaling Robot Learning | 20‘ + 10‘(Q&A) |
| 10:05 | 10:35 | Prof. Kris Hauser, University of Illinois Urbana-Champaign Title: Representation learning to interact with an uncertain world | 20‘ + 10‘(Q&A) |
| 10:35 | 10:45 | Coffee/Tea Break | 10‘ |
| 10:45 | 11:15 | Poster Teasers, Display and Discussions | 30‘ |
| 11:15 | 11:45 | Dr. Ted Xiao, Google DeepMind Title: What’s Missing for Robotics-First Foundation Models | 20‘ + 10‘(Q&A) |
| 11:45 | 12:15 | Prof. Tamim Asfour, Karlsruhe Institute of Technology (KIT) Title: Mobile Manipulation – Lessons Learned and Challenges to Overcome | 20‘ + 10‘(Q&A) |
| 12.15 | 13.00 | Panel Discussion & Best Poster Award Moderator: Dr. Rajkumar Muthusamy, Dubai Future Foundation | 45’ |
| 13.00 | 13.30 | DFL Sponsored Lunch + Networking | 30’ |
Submissions
————————————————————————————————————————————————————
1. Towards a Miniature Bi-Manual End-effector for Compact Coordinated Robot Dexterity
Sharfin Islam, Matei Ciocarlie
2. Uncertainty-Aware Map-Space Dynamics Models for Manipulation-Enhanced Mapping
Nils Dengler, Joao Marcos Correia Marques, Tobias Zaenker, Vamsi Kalagaturu, Shenlong Wang, Maren Bennewitz, Kris Hauser
3. Learning Goal-Directed Object Pushing in Cluttered Scenes with Location-Based Attention
Nils Dengler, Juan Del Aguila Ferrandis, João Moura, Sethu Vijayakumar, Maren Bennewitz
4. CuriousBot: Interactive Mobile Exploration via Actionable 3D Relational Object Graph
Yixuan Wang, Leonor Fermoselle, Tarik Kelestemur, Jiuguang Wang, Yunzhu Li
5. EdgeVLA: Efficient Vision-Language-Action Models
Paweł Budzianowski, Wesley Maa, Matthew Freed, Jingxiang Mo, Winston Hsiao, Aaron Xie, Tomasz Młoduchowski, Viraj Tipnis, Benjamin Bolte
6. A Novel Text-to-Action Model and Performance Evaluation of LLMs for Intention Detection in Pick and Place Tasks
Abhinav Pathak, V. Kalaichelvi
7. Dynamic Object Catching with Quadruped Robot Front Legs
André Schakkal, Guillaume Bellegarda, Auke Ijspeert
8. BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-Wide Mobile Manipulation
Rutav Shah, Albert Yu, Yifeng Zhu, Yuke Zhu, Roberto Martı́n-Martı́n
9. Open6DOR: Benchmarking Open-instruction 6-DoF Object Rearrangement and A VLM-based Approach
Yufei Ding, Haoran Geng, Chaoyi Xu, Xiaomeng Fang, Jiazhao Zhang, Songlin Wei, Qiyu Dai, Zhizheng Zhang, He Wang
10. Object Transportation on the Water Surface using a UAV-USV Team
Filip Novák, Tomáš Báča, Martin Saska
————————————————————————————————————————————————————
Call for Papers
We welcome authors to submit extended abstracts (2-4 pages excluding references) showcasing relevant work on workshop topics, including field-collected dataset analysis, robot technologies and platforms, interactive demonstrations, and innovative, creative, and futuristic solutions within the workshop context.
- Format: Submissions should be in IROS conference standard PDF format.
- Page Limit: Maximum of 2-4 pages (excluding references).
- Review Process: All submissions will undergo peer review based on relevance, technical quality, and ethics.
- Presentation Opportunities: Accepted papers will be presented as posters, with some selected for spotlight talks during the workshop.
- Publication: Accepted papers will be available on the workshop website.
Submission Timelines
**Submission deadline: 20th September 2024**
**Notification of acceptance: 22nd September 2024**
**Workshop date: 14/10/2024**
**Please send your contribution to: rajkumar.muthusamy@dubaifuture.gov.ae (Subject : IROS-MoMA3 Workshop Contribution | Corresponding Author Name | University/Industry Name)**
Awards
- Best Poster Award: 500$ USD
Endorsed By IEEE RAS Technical Committees
- Cognitive Robotics
- Mobile Manipulation
- Robot Learning
Sponsors
