Ruta Desai
I am a Staff Research Scientist in Fundamental AI Research (FAIR) at Meta. At FAIR, I am exploring reasoning and long-horizon planning problems towards agents that can partner with humans. Prior to that, I was a Tech Lead Manager at Meta Reality Labs Research. At Reality Labs, I led an embodied AI team focused on solving perception and planning problems for Augmented Reality (AR) systems that can extend their users' capabilities. I am broadly interested in perception, planning, and reasoning problems for Human-AI interaction.
I obtained my PhD at the Robotics Institute, Carnegie Mellon University in 2018 working with Stelian Coros and Jim McCann. For my thesis, I devised human-AI systems that enabled casual users to build and program robots — towards increasing accessibility of robotics. In the past, I have also had not-so-brief stints in the area of human motor control and biomechanical simulations with Hartmut Geyer and Jessica K. Hodgins.
Email: rutadesai[at]meta.com
Collaborative Reasoner: Self-improving Social Agents with Synthetic Conversations
Ansong Ni*, Ruta Desai*, Yang Li, Xinjie Lei, Dong Wang, Jane Yu, Ramya Raghavendra, Gargi Gosh, Daniel Li, Asli Celikyilmaz
[Under Review] Advances in Neural Information Processing Systems (Neurips), San Diego, USA (2025).
PDF | Code | Meta AI Blog
ADAPT: Actively Discovering and Adapting to Preferences for any Task
Maithili Patel, Xavier Puig, Ruta Desai, Roozbeh Mottaghi, Sonia Chernova, Joanne Truong, and Akshara Rai
[Under Review] Conference on Language Modeling (COLM), Montreal, Canada (2025).
PDF
Ask-to-Act: Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learning
Ram Ramrakhya, Matthew Chang, Xavier Puig, Ruta Desai, Zsolt Kira, and Roozbeh Mottaghi
[Under Review] International Conference on Computer Vision (ICCV), Hawaii, USA (2025).
PDF
PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks
Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander, Tsung-Yen Yang
International Conference on Learning Representations (ICLR), Singapore (2025).
PDF | Code | Webpage | Meta AI Blog
Learning Human Preferences Over Robot Behavior as Soft Planning Constraints
Austin Narcomey, Nathan Tsoi, Ruta Desai, and Marynel Vazquez
[Under Review] ACM Transactions on Human Robot Interaction (HRI) 2025.
arXiv
Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
Xavi Puig*, Eric Undersander*, Andrew Szot*, Mikael Cote*, Ruslan Partsey*, Jimmy Yang*, Ruta Desai*, Alexander Clegg*, Michal Hlavac, Tiffany Min, Theo Gervet, Vladimir Vondrus, Vincent-Pierre Berges, John Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai**, Roozbeh Mottaghi**
International Conference on Learning Representations (ICLR), Vienna, Austria (2024).
PDF | Code | Webpage
Adaptive Coordination in Social Embodied Rearrangement
Andrew Szot, Unnat Jain, Dhruv Batra, Zsolt Kira, Ruta Desai, and Akshara Rai
International Conference on Machine Learning (ICML), Hawaii, USA (2023).
PDF | Code
Effective Baselines for Multiple Object Rearrangement Planning in Partially Observable Mapped Environments
Engin Tekin, Elaheh Barati, Nitin Kamra, and Ruta Desai
AAAI Workshop on User-Centric AI for Assistance in At-Home Tasks, Washington DC, USA (2023).
arXiv | bib
Cross-Domain Imitation Learning via Semantic Skills
Karl Pertsch, Ruta Desai, Franziska Meier, Vikash Kumar, Dhruv Batra, and Akshara Rai
Conference on Robot Learning(CoRL), Aukland, New Zealand (2022).
PDF | bib | Video | Webpage
User-in-the-loop Evaluation of Multimodal LLMs for Activity Assistance
Mrinal Verghese*, Brian Chen*, Hamid Eghbalzadeh, Tushar Nagarajan, and Ruta Desai
IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Arizona, USA. (2025).
PDF | Video
Pretrained Language Models as Visual Planners for Human Assistance
Dhruvesh Patel, Hamid Engbalzadeh, Nitin Kamra, Michael Louis Iuzzolino, Unnat Jain, and Ruta Desai
International Conference on Computer Vision (ICCV), Paris, France (2023).
PDF | Code
EgoTV: Egocentric Task Verification from Natural Language Task Descriptions
Rishi Hazra, Brian Chen, Akshara Rai, Nitin Kamra, and Ruta Desai
International Conference on Computer Vision (ICCV), Paris, France (2023).
PDF | Code | Webpage
Action Dynamics Task Graphs for Learning Plannable Representations of Procedural Tasks
Weichao Mao, Ruta Desai, Michael Louis Iuzzolino, and Nitin Kamra
AAAI Workshop on User-Centric AI for Assistance in At-Home Tasks, Washington DC, USA (2023).
arXiv | bib
Egocentric Scene Context for Human-centric Environment Understanding from Video
Tushar Nagarajan, Santhosh Kumar Ramakrishnan, Ruta Desai, James Hillis, and Kristen Grauman
Advances in Neural Information Processing Systems (Neurips), New Orleans, USA (2023).
PDF | bib | Webpage
Episodic Memory Question Answering
Samyak Datta, Sameer Dharur, Vincent Cartellier, Ruta Desai, Mukul Khanna, Dhruv Batra, and Devi Parikh
The Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA (2022).
PDF | bib | Video
How You Move Your Head Tells What You Do: Self-supervised Video Representation Learning with Egocentric Cameras and IMU Sensors
Satoshi Tsutsui, Ruta Desai, and Karl Ridgeway
International Conference on Computer Vision (ICCV), EPIC Workshop (2021).
PDF | bib | Video
A Meta-Bayesian Approach for Rapid Online Parametric Optimization for Wrist-based Interactions
Yi-Chi Liao, Ruta Desai, Alec M Pierce, Krista E Taylor, Hrvoje Benko, Tanya R Jonker, Aakar Gupta
ACM Conference on Human Factors in Computing Systems (CHI), Hawaii, USA (2024).
PDF | Video
Optimizing the Timing of Intelligent Suggestion in Virtual Reality
Difeng Yu, Ruta Desai, Ting Zhang, Hrvoje Benko, Tanya R. Jonker, and Aakar Gupta
ACM User Interface Software and Technology Symposium (UIST), Bend, USA (2022).
PDF | bib | Video
Optimal Assistance for Object-Rearrangement Tasks in Augmented Reality
Benjamin Newman, Kevin Carlberg, and Ruta Desai
arXiv (2020).
PDF | bib | Video
Towards Inferring Cognitive State Changes from Pupil Size Variations in Real World Conditions
Naga Venkata Kartheek Medathati, Ruta Desai, and James Hillis
ACM Symposium on Eye Tracking Research and Applications (ETRA), Stuttgart, Germany (2020).
PDF | bib
Geppetto: Enabling Semantic Design of Expressive Robot Behaviors
Ruta Desai, Fraser Anderson, Justin Matejka, Stelian Coros, James McCann, George Fitzmaurice, and Tovi Grossman
ACM Conference on Human Factors in Computing Systems (CHI), Glasgow, UK (2019).
Best paper award (top 1%) [Details]
PDF | bib | Video | Fastforward | Supplementary (PDF)
Assembly-aware Design of Printable Electromechanical Devices
Ruta Desai, James McCann, and Stelian Coros
ACM User Interface Software and Technology Symposium (UIST), Berlin, Germany (2018).
PDF | bib | Video | Fastforward
Skaterbots: Optimization-based Design and Motion Synthesis for Robotic Creatures with Legs and Wheels
Moritz Geilinger, Roi Poranne, Ruta Desai, Bernhard Thomaszewski, and Stelian Coros
ACM Transactions on Graphics (Proc. ACM SIGGRAPH), Vancouver, Canada (2018).
PDF | bib | Video
Automatic Design of Task-specific Robotic Arms
Ruta Desai, Margarita Safonova, Katharina Muelling, and Stelian Coros
ICRA Workshop on Autonomous Robot Design, Brisbane, Australia (2018).
arXiv | bib | Slides | Video
Interactive Co-Design of Form and Function for Legged Robots using the Adjoint Method
Ruta Desai, Beichen Li, Ye Yuan, and Stelian Coros
International Conference on Climbing and Walking Robots (CLAWAR), Panama city, Panama (2018).
Best technical paper (second place)
arXiv Preprint | Video
Computational Abstractions for Interactive Design of Robotic Devices
Ruta Desai, Ye Yuan, and Stelian Coros
IEEE International Conference on Robotics and Automation (ICRA), Singapore (2017).
PDF | bib | Slides | Video
| Code
Robot models for fabrication: car, walking robot
3D Printing Pneumatic Device Controls with Variable Activation Force Capabilities
Marynel Vazquez, Eric Brockmeyer, Ruta Desai, Chris Harrison and Scott E. Hudson
ACM Conference on Human Factors in Computing Systems (CHI), Seoul, Korea, (2015).
PDF | bib | Poster | Video
Human Action Anticipation: A Survey
Bolin Lai*, Sam Toyer*, Tushar Nagarajan, Rohit Girdhar, Shengxin Zha, James M. Rehg, Kris Kitani, Kristen Grauman, Ruta Desai, and Miao Liu
arXiv Preprint (2024).
Human-Centered Planning
Yuliang Li, Nitin Kamra, Ruta Desai, and Alon Halevy
arXiv Preprint (2023).
Optimal Assistance for Object-Rearrangement Tasks in Augmented Reality
Benjamin A. Newman, Kevin T. Carlberg, Ruta Desai, James Hillis
US Patent US20220114366 A1 (2022).
Generative design techniques for robot behavior
Fraser Anderson, Stelian Coros, Ruta Desai, Tovi Grossman, Justin Matejka, George Fitzmaurice
US Patent 20200030988 A1 (2020).
Robust swing leg controller under large disturbances
Ruta Desai and Hartmut Geyer
US Patent 20150066156 A1 (2014).
A Brief Overview of Human and Robot Motor Learning
Ruta Desai
RI CMU Technical Report
I am always looking to support and contribute towards encouraging women in science and technology. In the past:
Last updated: May 2025
Webpage adapted from Jon Barron.