Jay Vakil

Beyond the End-Effector: Learning Whole-Body Peripersonal Representations for Collision-Aware Manipulation

Jay Vakil, Amine Boubrit, Artemis Shaw, Ekshan Raj Verma, Caleb Escobedo, Anuj Pasricha, Carson Kohlbrenner, Qinzhengfang Li, Alessandro Roncone

Dynamics-Compliant Trajectory Diffusion

Payload-conditioned diffusion model for generating dynamic motions for robots handling up to 3× nominal payloads.

Anuj Pasricha, Joewie J. Koh, Jay Vakil, Alessandro Roncone

Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Scaling robotic learning via the RT-X model and a massive cross-platform dataset of 60+ robots. The ‘ImageNet’ moment for robotics.

Open X-Embodiment Collaboration, including Jay Vakil

RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking

Universal robot learning with semantic augmentations. Teaching robots broadly generalizable skills through action chunking across 38+ tasks.

Homanga Bharadhwaj*, Jay Vakil*, Mohit Sharma*, Abhinav Gupta, Shubham Tulsiani, Vikash Kumar

OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics

A zero-shot system for open-vocabulary mobile manipulation. Integrating vision-language models with classical navigation primitives to handle messy environments.

Peiqi Liu*, Yaswanth Orru*, Jay Vakil, Chris Paxton, Nur Muhammad “Mahi” Shafiullah†, Lerrel Pinto†

What do we learn from a large-scale study of pre-trained visual representations in sim and real environments?

A large-scale study of pre-trained visual representations in sim and real environments to understand transfer efficacy.

Sneha Silwal*, Karmesh Yadav*, Tingfan Wu*, Jay Vakil*, Arjun Majumdar*, Sergio Arnaud*, Claire Chen, Vincent-Pierre Berges, Dhruv Batra, Aravind Rajeswaran, Mrinal Kalakrishnan, Franziska Meier, Oleksandr Maksymets

Towards Open-World Mobile Manipulation in Homes: Lessons from the NeurIPS 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge

Open-vocabulary mobile manipulation challenge. Benchmarking robot performance in novel home environments.

Sriram Yenamandra*, Arun Ramachandran*, Mukul Khanna*, Karmesh Yadav*, Jay Vakil*, Andrew Melnik, Michael Büttner, Leon Harz, Lyon Brown, Gora Chand Nandi, Arjun PS, Gaurav Kumar Yadav, Rahul Kala, Robert Haschke, Yang Luo, Jinxin Zhu, Yansen Han, Bingyi Lu, Xuan Gu, Qinyuan Liu, Yaping Zhao, Qiting Ye, Chenxiao Dou, Yansong Chua, Volodymyr Kuzma, Vladyslav Humennyy, Ruslan Partsey, Jonathan Francis, Devendra Singh Chaplot, Gunjan Chhablani, Alexander Clegg, Theophile Gervet, Vidhi Jain, Ram Ramrakhya, Andrew Szot, Austin Wang, Tsung-Yen Yang, Aaron Edsinger, Charlie Kemp, Binit Shah, Zsolt Kira, Dhruv Batra, Roozbeh Mottaghi, Yonatan Bisk, Chris Paxton

Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?

Evaluating mid-level visual representations for robotic control tasks. Are we closer to an artificial visual cortex?

Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Tingfan Wu, Jay Vakil, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, Franziska Meier

RoboHive: A Unified Framework for Robot Learning

A unified framework for robot learning and embodied AI research, offering diverse simulation environments.

Vikash Kumar, Rutav Shah, Gaoyue Zhou, Vincent Moens, Vittorio Caggiano, Jay Vakil, Abhishek Gupta, Aravind Rajeswaran

Spatial-Language Attention Policies for Efficient Robot Learning

Efficient robot learning using spatial-language attention mechanisms for manipulation tasks.

Priyam Parashar, Vidhi Jain, Xiaohan Zhang, Jay Vakil, Sam Powers, Chris Paxton