[RSS 2021] An End-to-End Differentiable Framework for Contact-Aware Robot Design

Overview

DiffHand

This repository contains the implementation for the paper An End-to-End Differentiable Framework for Contact-Aware Robot Design (RSS 2021).

In this paper, we propose a fully differentiable pipeline to jointly optimize the morphology and control of manipulator robots. At the core of the framework is a deformation-based morphology parameterization and a differentiable simulation.

The framework itself is general and not limited to manipulator robots, we select the case study of manipulator robots because of its complexity and contact-rich nature. Welcome to try our code on any other types robots as well.

teaser

Installation

We provides two methods for installation of the code. Install on local machine and Install by Docker.

Option 1: Install on Local Machine

Operating System: tested on Ubuntu 16.04 and Ubuntu 18.04

  1. Clone the project from github: git clone https://github.com/eanswer/DiffHand.git --recursive .

  2. Install CMake >= 3.1.0: official instruction for cmake installation

  3. build conda environment and install simulation

    cd DiffHand
    conda env create -f environment.yml
    conda activate diffhand
    cd core
    python setup.py install
    
  4. Test the installation

    cd examples
    python test_redmax.py
    

    If you see a simulation rendering with a two-link pendulum as below, you have successfully installed the code base.

    test_redmax

Option 2: Install by Docker

We provide a docker installation in the docker folder. Follow the readme instruction in docker folder to complete the installation.

Code Structure

There are two main components of the code base:

  • Differentiable RedMax: DiffHand/core. The differentiable redmax is based off RedMax and further makes if fully differentiable. It provides the simulation derivatives w.r.t. both simulation parameters (kinematics- and dynamics-related parameter) and control actions. It is implemented in C++ for computing efficiency. We provide a simulation document for mathematical details of our differentiable RedMax.
  • Morphology and Control Co-Optimization: DiffHand/examples. We build an end-to-end differentiable framework to co-optimize both the morphology and control of manipulators. We use L-BFGS-B as our default gradient-based optimizer and also provides the source code for the gradient-free baseline methods.

Run the Code

It is recommended to try out the scripts in play with redmax simulation first if you would like to get familiar with simulation interface.

Run the examples in the paper

We include the four co-design tasks from the paper in the examples folder.

  • Finger Reach
  • Rotate Cube
  • Flip Box
  • Assemble

To run the L-BFGS-B optimization with our deformation-based design parameterization, you can enter the corresponding folder and run demo.sh under the folder. For example, to run Finger Reach,

cd examples/rss_finger_reach
bash demo.sh

Run batch experiments of baseline algorithms

We include the gradient-free baselines (except RL) and the control-only baseline in this repository. For the RL baseline, we use the released code from Luck et al with some modifications to our proposed morphology parameterization.

To run the baseline algorithms or our method in a batch mode, enter the corresponding folder and run run_batch_experiments.py. For example, to run Flip Cube with CMA-ES,

cd examples/rss_finger_flip
python run_batch_experiments.py --method CMA --num-seeds 5 --num-processes 5 --save-dir ./results/

Play with redmax simulation

We provide several examples to test the forward simulation and its differentiability.

  • examples/test_redmax.py provides the script to show how to run forward simulation and rendering. It can be easily executed by:

    python test_redmax.py --model hand_sphere
    

    Here, you can also try other models provided in assets folder (models are described by xml configuration files).

  • examples/test_finger_flick_optimize.py provides an example for using the backward gradients of the simulation. In this example, we use gradient-based optimization to optimize the control sequence of a pendulum finger model to flick a cube to a target location. run it by:

    python test_finger_flick_optimize.py
    

    The initial control sequence is shown first and you can press [Esc] to close the rendering and start the optimization. After successful optimization, you will see a rendering as below:

    finger_flick

Citation

If you find our paper or code is useful, please consider citing:

@INPROCEEDINGS{Xu-RSS-21, 
    AUTHOR    = {Jie Xu AND Tao Chen AND Lara Zlokapa AND Michael Foshey AND Wojciech Matusik AND Shinjiro Sueda AND Pulkit Agrawal}, 
    TITLE     = {{An End-to-End Differentiable Framework for Contact-Aware Robot Design}}, 
    BOOKTITLE = {Proceedings of Robotics: Science and Systems}, 
    YEAR      = {2021}, 
    ADDRESS   = {Virtual}, 
    MONTH     = {July}, 
    DOI       = {10.15607/RSS.2021.XVII.008} 
} 
This robot lcoalisation package for lidar-map based localisation using multi-sensor state estimation.

A ROS-based NDT localizer with multi-sensor state estimation This repo is a ROS based multi-sensor robot localisation. An NDT localizer is loosely-cou

null 39 Jun 6, 2022
This is a group project for the unit Technical Software Design.

electoral-project This is a group project for the unit Technical Software Design. Group number: 9 Members of this group: Grace Tang, Lorien Cutler, Jo

Grace Tang 3 May 24, 2021
SSL_SLAM2: Lightweight 3-D Localization and Mapping for Solid-State LiDAR (mapping and localization separated) ICRA 2021

SSL_SLAM2 Lightweight 3-D Localization and Mapping for Solid-State LiDAR (Intel Realsense L515 as an example) This repo is an extension work of SSL_SL

Wang Han 王晗 312 Jul 1, 2022
Caffe: a fast open framework for deep learning.

Caffe Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR)/The Berke

Berkeley Vision and Learning Center 32.7k Jun 24, 2022
R2LIVE is a robust, real-time tightly-coupled multi-sensor fusion framework, which fuses the measurement from the LiDAR, inertial sensor, visual camera to achieve robust, accurate state estimation.

R2LIVE is a robust, real-time tightly-coupled multi-sensor fusion framework, which fuses the measurement from the LiDAR, inertial sensor, visual camera to achieve robust, accurate state estimation.

HKU-Mars-Lab 568 Jun 21, 2022
A Modular Framework for LiDAR-based Lifelong Mapping

LT-mapper News July 2021 A preprint manuscript is available (download the preprint). LT-SLAM module is released.

Giseop Kim 209 Jun 23, 2022
Ingescape - Model-based framework for broker-free distributed software environments

Ingescape - Model-based framework for broker-free distributed software environments Overview Scope and Goals Ownership and License Dependencies with o

The ZeroMQ project 25 Jun 10, 2022
Machine Learning Framework for Operating Systems - Brings ML to Linux kernel

Machine Learning Framework for Operating Systems - Brings ML to Linux kernel

File systems and Storage Lab (FSL) 176 Jun 27, 2022
Dummy-Robot my super mini robot arm robot items

Dummy-Robot 我的超迷你机械臂机器人项目。 资料待整理 已添加3D模型设计源文件。 已添加夹爪硬件设计文件和LED灯环PCB 已添加无线空间定位控制器PCB文件 已添加无线示教器Peak软硬件工程(作为submodule) 已添加REF的硬件设计文件 已添加DummyStudio上位机 已

稚晖 7.6k Jun 27, 2022
End to end test framework designed for Juce applications

JUCE End to End test framework What is it? This package provides a mechanism to end-to-end test a JUCE application Prerequisites CMake. Must be 3.18 o

Focusrite Audio Engineering Ltd. 36 May 29, 2022
Official PyTorch Code of GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection (CVPR 2021)

GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Mo

Abhinav Kumar 73 Jun 22, 2022
[WIP] Differentiable Simulation of Soft Multi-body Systems (NeurIPS 2021)

Differentiable Simulation of Soft Multi-body Systems Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, Ming C. Lin [Paper] [Code] Updates The C++ backend s

YilingQiao 22 May 31, 2022
Convert YouTube Subscriptions JSON into RSS Reader Compatible OPML

OPMLify Convert YouTube Subscriptions JSON into RSS Reader Compatible OPML Brief Overview OPMLify allows you to import your YouTube Subscriptions to a

null 13 May 25, 2022
Subscribe anime RSS to download, repack, upload to Baidu, and post on TSDM automatically.

AnimeRSSforTSDM This is a project to automatically transfer Anime to Baidu netdisk with some rules on TSDM. Watch this post in detail. System Flow Cha

InanitySnow 2 May 15, 2022
Multi-Joint dynamics with Contact. A general purpose physics simulator.

Multi-Joint dynamics with Contact. A general purpose physics simulator.

DeepMind 4.5k Jun 28, 2022
Source Codes for Codimensional Incremental Potential Contact (C-IPC)

Source Codes for Codimensional Incremental Potential Contact (C-IPC) Reference This repository provides source code for: Minchen Li, Danny M. Kaufman,

null 86 Jun 22, 2022
Source Codes for Injective Deformation Processing (IDP) with Incremental Potential Contact (IPC)

Source Codes for Injective Deformation Processing Reference This repository provides source code for: Yu Fang*, Minchen Li* (equal contribution), Chen

null 33 Jun 22, 2022
Vulkan Video Sample Application demonstrating an end-to-end, all-Vulkan, processing of h.264/5 compressed video content.

This project is a Vulkan Video Sample Application demonstrating an end-to-end, all-Vulkan, processing of h.264/5 compressed video content. The application decodes the h.264/5 compressed content using an HW accelerated decoder, the decoded YCbCr frames are processed with Vulkan Graphics and then presented via the Vulkan WSI.

NVIDIA DesignWorks Samples 108 Jun 18, 2022
WeNet is to close the gap between research and production end-to-end (E2E) speech recognition models,

WeNet is to close the gap between research and production end-to-end (E2E) speech recognition models, to reduce the effort of productionizing E2E models, and to explore better E2E models for production.

null 2.2k Jun 29, 2022
BladeDISC - BladeDISC is an end-to-end DynamIc Shape Compiler project for machine learning workloads.

BladeDISC Introduction Overview Features and Roadmap Frontend Framework Support Matrix Backend Support Matrix Deployment Solutions Numbers of Typical

Alibaba 376 Jun 24, 2022
LLVM IR and optimizer for shaders, including front-end adapters for GLSL and SPIR-V and back-end adapter for GLSL

Licensing LunarGLASS is available via a three clause BSD-style open source license. Goals The primary goals of the LunarGLASS project are: Reduce the

LunarG, Inc. 151 Jun 18, 2022
Khronos-reference front end for GLSL/ESSL, partial front end for HLSL, and a SPIR-V generator.

News Visual Studio 2013 is no longer supported As scheduled, Microsoft Visual Studio 2013 is no longer officially supported. Please upgrade to at leas

The Khronos Group 2.2k Jun 25, 2022
WIP: ESP32 powered robot dog, quadruped robot. This is just code, hardware in the other repositories

Small Robot dog (quadruped) Hardware ESP32 IMU (not implemented) 12 servos TowerPro mg90d (hope it will work) Two 18650 Software Arduino IDE compatibl

Gleb Devyatkin 60 Jun 8, 2022
A ROS robot supporting voice control, autonomous navigation and robot arm motion.

ROS Service Robot Project Due to the GitLab restrictions, only screenshot is displayed here. Warehouse Overview robot: The development source code is

SilenceJiang 56 Apr 12, 2022
Compiler Design Project: Simulation of front-end phase of C Compiler involving switch-case construct.

CSPC41 Compiler Design Project Assignment Compiler Design Project: Simulation of front-end phase of C Compiler involving switch-case construct. Using

Adeep Hande 1 Dec 15, 2021
Source Code and Embedded Design of Our Factory Robot at AUTCup 2015 Competitions

AUTCup 2015 @Factory Robot This repository contains the source code and embedded design of our @Factory robot at AUTCup 2015 competitions. The robot w

Meysam Parvizi 1 Oct 20, 2021
Fast, differentiable sorting and ranking in PyTorch

Torchsort Fast, differentiable sorting and ranking in PyTorch. Pure PyTorch implementation of Fast Differentiable Sorting and Ranking (Blondel et al.)

Teddy Koker 587 Jun 22, 2022
ADOP: Approximate Differentiable One-Pixel Point Rendering

ADOP: Approximate Differentiable One-Pixel Point Rendering

Darius Rückert 1.8k Jun 24, 2022
Simple, differentiable rigid body dynamics

Quark Quark is intended to be a simple differentiable rigid body dynamics library. For the time being it is starting out as a ground-up rewrite of RBD

Pedro Morais 4 Nov 23, 2021