Nimble: Physics Engine for Deep Learning

Overview

Stanford Nimble Logo

Build Status

Stanford Nimble

pip3 install nimblephysics

** BETA SOFTWARE **

Use physics as a non-linearity in your neural network. A single timestep, nimble.timestep(state, controls), is a valid PyTorch function.

Forward pass illustration

We support an analytical backwards pass, that works even through contact and friction.

Backpropagation illustration

It's as easy as:

from nimble import timestep

# Everything is a PyTorch Tensor, and this is differentiable!!
next_state = timestep(world, current_state, control_forces)

Nimble started life as a fork of the popular DART physics engine, with analytical gradients and a PyTorch binding. We've worked hard to maintain as much backwards compatability as we can, so many simulations that worked in DART should translate directly to Nimble.

Check out our website for more information.

Issues
  • Issue with Rajagopal example

    Issue with Rajagopal example

    Hi!

    Thanks a lot for the amazing work! When I try to run the rajagopal.py example, I encountered the error below:

    ❯ python rajagopal.py
    Msg [NameManager::issueNewName] (default) The name [Joint_rot_z] is a duplicate, so it has been renamed to [Joint_rot_z(1)]
    Msg [NameManager::issueNewName] (default) The name [hip_r_z] is a duplicate, so it has been renamed to [hip_r_z(1)]
    Msg [NameManager::issueNewName] (default) The name [hip_l_z] is a duplicate, so it has been renamed to [hip_l_z(1)]
    Msg [NameManager::issueNewName] (default) The name [back_z] is a duplicate, so it has been renamed to [back_z(1)]
    Msg [NameManager::issueNewName] (default) The name [acromial_r_z] is a duplicate, so it has been renamed to [acromial_r_z(1)]
    Msg [NameManager::issueNewName] (default) The name [acromial_l_z] is a duplicate, so it has been renamed to [acromial_l_z(1)]
    Traceback (most recent call last):
      File "rajagopal.py", line 12, in <module>
        world.addSkeleton(skel)
    TypeError: addSkeleton(): incompatible function arguments. The following argument types are supported:
        1. (self: nimblephysics_libs._nimblephysics.simulation.World, skeleton: nimblephysics_libs._nimblephysics.dynamics.Skeleton) -> str
    
    Invoked with: <nimblephysics_libs._nimblephysics.simulation.World object at 0x7f95df2df880>, <nimblephysics_libs._nimblephysics.biomechanics.OpenSimFile object at 0x7f95df2df8b8>
    

    I have the version 0.6.1 installed via pip. How can I solve this problem?

    opened by AlleUndKalle 3
  • How to reset the joint DOF

    How to reset the joint DOF

    Hi,

    I am loading a humanoid from a urdf file. It seems that we can not directly define a ball joint in urdf. So I wonder if we can load a urdf with 1 DOF and then change the DOF to 3 after loading the model into the nimble?

    Thanks

    opened by zhangy76 2
  • Joint degree of freedom

    Joint degree of freedom

    Hi,

    The object is described via generalized coordinate. I wonder how to get the joint DOF of each link? The question may be very basic... So I wonder if there is any documents besides the examples in the current repository that introduce the basic functions?

    Thanks

    opened by zhangy76 2
  • getMultipleContactInverseDynamicsOverTime

    getMultipleContactInverseDynamicsOverTime

    Hi,

    I am able to run all the examples related to the inverse dynamics but fail to call the "getMultipleContactInverseDynamicsOverTime" function. Specifically, I input the arguments following the requirements but the program still indicates the input format is wrong. So I am wondering if anyone has use this function successfully ?

    Thanks

    opened by zhangy76 1
  • segmentation fault when calling `getJoints()`

    segmentation fault when calling `getJoints()`

    The following is the behavior that I noticed in Python:

        world = nimble.simulation.World()
        skel: nimble.dynamics.Skeleton = world.loadSkeleton(
            "data/sdf/atlas/ground.urdf"
        )
        skel.getJoints()
    
    opened by michguo 1
  • `integrateVelocitiesFromImpulses` and `integratePositions`

    `integrateVelocitiesFromImpulses` and `integratePositions`

    Round 3 Review

    • Add default arg for integrateVelocitiesFromImpulses.

    Round 2 Review

    • Add python bindings.

    Round 1 Review

    • Separate position integration from impulse velocity integration.
    • Create functions for velocity (impulse) integration and position integration.
    opened by michguo 1
  • Drop frames in excess of 100fps on C++ end, before hitting web GUI

    Drop frames in excess of 100fps on C++ end, before hitting web GUI

    If we try to display a simulation from 1000fps at real time in the browser, right now the C++ will blindly attempt to send JSON packets to the browser at 1000fps. That's too much for the browser to handle, and slows everything down. So we should really have the C++ rate-limit itself to 100fps, and silently drop/batch updates that come in faster than that on the C++ side.

    enhancement 
    opened by keenon 1
  • dart_layer needs update setForces -> setExternalForces

    dart_layer needs update setForces -> setExternalForces

    File "/home/yannis/anaconda3/envs/pytorch/lib/python3.7/site-packages/diffdart/dart_layer.py", line 96, in dart_layer return DartLayer.apply(world, pos, vel, torque, pointer) # type: ignore File "/home/yannis/anaconda3/envs/pytorch/lib/python3.7/site-packages/diffdart/dart_layer.py", line 35, in forward world.setForces(torque.detach().numpy()) AttributeError: 'diffdart_libs._diffdart.simulation.World' object has no attribute 'setForces'

    opened by iexarchos 1
  • renderTrajectoryLines produces jittery lines

    renderTrajectoryLines produces jittery lines

    If you call gui.stateMachine().renderTrajectoryLines(...) multiple times with the same input, you'll get different results visually. This indicates to me some kind of data corruption or race conditions hidden in here that we need to ferret out.

    opened by keenon 1
  • Capsule-Floor penetration

    Capsule-Floor penetration

    We're seeing penetration in Yannis's demos: https://github.com/iexarchos/DiffDART_DDP-iLQR_opt

    Specifically, HalfCheetah and Reacher

    This is probably an issue with capsule-box collisions.

    opened by keenon 1
  • Support differentiating through

    Support differentiating through "Constraint Force Mixing"

    Our LCP's are only guaranteed solvable if A is positive-semidefinite and there are no force bounds. To increase stability of our LCPs, we can do a trick called "Constraint Force Mixing". In practice, this means multiplying the elements on the diagonal of A by 1.0 + eps, where eps is some small positive value. This has the effect of ensuring A isn't singular, and in general reducing "singular-ness" of A.

    You can turn constraint force mixing on and off with void World::setConstraintForceMixingEnabled(bool enable). Currently, by default CFM is turned off, because our Jacobians don't support it.

    This ticket is about supporting it.

    The matrix A is computed in dart/constrant/BoxedLcpConstraintSolver.cpp in the method solveConstrainedGroup(). The A matrix it computes is in Open Dynamics Engine format, which means row-major order, where each row's length is rounded up to the nearest multiple of 4 (to allow vectorization) and any padding entries are ignored. This calls out to individual constraints to populate A. For each constraint, it applies a unit impulse, and then measures the change in relative velocity at the constraint. The method that applies the CFM is ContactConstraint::getVelocityChange(), towards the bottom.

    Supporting this in our differentiation means tracking all the CFM constants for each element of the diagonal of A, and storing them for later. These are constants wrt differentation, but we need to ensure that we scale A's diagonals, and the gradient of A's diagonals, by these constants wherever it is computed.

    opened by keenon 1
  • Body node setScale() leads to incorrect origin of the child link

    Body node setScale() leads to incorrect origin of the child link

    Bug Report

    Environment

    Ubuntu 18.04, GCC 7.4.0

    Expected Behavior

    Current Behavior

    When I use setScale(), only the shape of the current node is scaled while the origin of the child link becomes locating at the zeros.

    Steps to Reproduce

    Code to Reproduce

    robot_path = "~.urdf" world: nimble.simulation.World = nimble.simulation.World() skel: nimble.dynamics.Skeleton = world.loadSkeleton(robot_path) world.getBodyNodeByIndex(node).setScale([1, 0.8, 1])

    opened by zhangy76 0
  • add getLocalVertices()

    add getLocalVertices()

    [Remove this line and describe this pull request. Link to relevant GitHub issues, if any.]


    Before creating a pull request

    • [ ] Document new methods and classes
    • [ ] Format new code files using clang-format

    Before merging a pull request

    • [ ] Set version target by selecting a milestone on the right side
    • [ ] Summarize this change in CHANGELOG.md
    • [ ] Add unit test(s) for this change
    opened by michguo 0
  • `mguo/my-master-20220324`

    `mguo/my-master-20220324`

    Steps to reproduce:

    # Checkout the lastest tested commit on the master branch.
    git checkout master
    git checkout 5429688496ce8094b9da01f4867b1d8ceaa1c330
    
    # Branch off of master
    git checkout -b mguo/my-master-20220324
    
    # Merge in the branches we care about.
    git merge mguo/impulses
    git merge mguo/lcp-result
    # Note: when you merge in `mguo/lcp-result`, there will be 3 merge conflicts across the 3 the `BoxedLcpConstraintSolver` files. For all conflicts, accept the incoming changes (a.k.a. changes from `mguo/lcp-result`).
    
    git merge mguo/lcp-utils
    git merge mguo/collision-group
    git merge mguo/body-node
    

    Build Variant: Release.

    opened by michguo 0
  • return `LcpResult` in `solveLcp`

    return `LcpResult` in `solveLcp`

    Defines a struct that solveLcp() returns called LcpResult, and stores all useful information from solveLcp() inside of this struct.

    Notes:

    • This branch should only be merged after merging in #72. The reason is that #72 changes the type of impulses from a vector to a matrix, and this PR assumes that the type of impulses is a matrix. Without merging in #72, your code will not build successfully.
    opened by michguo 0
  • Add python bindings for `LCPUtils`

    Add python bindings for `LCPUtils`

    1. Create constraint.LCPUtils.cpp for python bindings.
    2. Add python binding for isLCPSolutionValid().
    3. Implement getLCPSolutionType(), which returns the type of failure if the LCP solution failed.
    4. Implement LCPSolutionType enum.
    opened by michguo 0
Releases(v0.8.4)
  • v0.8.4(Jun 27, 2022)

  • v0.8.3(Jun 8, 2022)

    This adds MarkerFitter::generateDataErrorsReport(...) and MarkerFitter::checkForFlippedMarkers(...), which can be used to find and fix mislabeled markers (which are alarmingly common in published biomechanics datasets).

    Source code(tar.gz)
    Source code(zip)
  • v0.8.2(Jun 3, 2022)

  • v0.8.1(Jun 1, 2022)

  • v0.8.0(May 31, 2022)

    The old JSON-based protocol for storing trajectories for the GUI was reaching hundreds of MBs per trajectory. This new protobuf based approach cuts that down to single digit MBs, which zip up even smaller.

    Source code(tar.gz)
    Source code(zip)
  • v0.7.7(May 26, 2022)

  • v0.7.6(May 25, 2022)

    There are a bunch of user-input points in biomechanics where people can make mistakes, and we've added code to fix those in this release:

    • Fix CustomJoints that take offset into themselves at their default point, rather than in their distance to their parent or child.
    • Fix C3D files with markers that flip
    • Fix C3D files with a different coordinate system (different opinion about whether Y or Z means "up")
    Source code(tar.gz)
    Source code(zip)
  • v0.7.5(May 19, 2022)

  • v0.7.4(May 18, 2022)

    Adding the ability to auto-rotate *.mot files into the same coordinate system as we used to load a C3D file, so that marker RMSE comparisons are correct.

    Source code(tar.gz)
    Source code(zip)
  • v0.7.3(May 16, 2022)

  • v0.7.2(May 13, 2022)

  • v0.7.1(May 12, 2022)

    IKErrorReport now has getSortedMarkerRMSE() to return a list of marker errors, sorted from highest to lowest RMSE, and saveCSVMarkerErrorReport(path), which writes out a file with timestep-by-timestep errors for each marker

    Source code(tar.gz)
    Source code(zip)
  • v0.7.0(May 11, 2022)

  • v0.6.22.1(May 11, 2022)

  • v0.6.22(May 11, 2022)

  • v0.6.21(May 11, 2022)

  • v0.6.20(May 6, 2022)

  • v0.6.19(May 4, 2022)

  • v0.6.18(May 4, 2022)

  • v0.6.17(Apr 28, 2022)

  • v0.6.15.1(Mar 4, 2022)

  • v0.6.15(Mar 4, 2022)

    Fixing a pair of errors:

    • Fix handling for C3D files with NaN marker positions (those observations are now dropped)
    • Fix joint center and joint axis solvers, they are now robust to neighboring markers which do not appear at all in the trajectory.
    Source code(tar.gz)
    Source code(zip)
  • v0.6.14(Feb 25, 2022)

  • v0.6.12(Feb 23, 2022)

  • v0.6.11(Feb 22, 2022)

  • v0.6.10(Feb 21, 2022)

  • v0.6.8(Feb 11, 2022)

    The multi-trial autoscaling in MarkerFitter now downsamples the input if it's faced with too many trials at once, or too many total frames of input.

    Source code(tar.gz)
    Source code(zip)
  • v0.6.7(Feb 9, 2022)

    The biomechanics.MarkerFitter object now supports jointly scaling over multiple trials simultaneously, to support that feature in BiomechNet.

    Source code(tar.gz)
    Source code(zip)
A library for creating Artificial Neural Networks, for use in Machine Learning and Deep Learning algorithms.

iNeural A library for creating Artificial Neural Networks, for use in Machine Learning and Deep Learning algorithms. What is a Neural Network? Work on

Fatih Küçükkarakurt 5 Apr 5, 2022
PPLNN is a high-performance deep-learning inference engine for efficient AI inferencing.

PPLNN, which is short for "PPLNN is a Primitive Library for Neural Network", is a high-performance deep-learning inference engine for efficient AI inferencing.

null 814 Jun 19, 2022
Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.

This is the Vowpal Wabbit fast online learning code. Why Vowpal Wabbit? Vowpal Wabbit is a machine learning system which pushes the frontier of machin

Vowpal Wabbit 8k Jul 1, 2022
Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

Apache MXNet (incubating) for Deep Learning Apache MXNet is a deep learning framework designed for both efficiency and flexibility. It allows you to m

The Apache Software Foundation 20k Jul 3, 2022
Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit

CNTK Chat Windows build status Linux build status The Microsoft Cognitive Toolkit (https://cntk.ai) is a unified deep learning toolkit that describes

Microsoft 17.2k Jun 24, 2022
header only, dependency-free deep learning framework in C++14

The project may be abandoned since the maintainer(s) are just looking to move on. In the case anyone is interested in continuing the project, let us k

tiny-dnn 5.5k Jun 22, 2022
LibDEEP BSD-3-ClauseLibDEEP - Deep learning library. BSD-3-Clause

LibDEEP LibDEEP is a deep learning library developed in C language for the development of artificial intelligence-based techniques. Please visit our W

Joao Paulo Papa 18 Mar 15, 2022
Caffe: a fast open framework for deep learning.

Caffe Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR)/The Berke

Berkeley Vision and Learning Center 32.7k Jul 1, 2022
Deep Learning API and Server in C++11 support for Caffe, Caffe2, PyTorch,TensorRT, Dlib, NCNN, Tensorflow, XGBoost and TSNE

Open Source Deep Learning Server & API DeepDetect (https://www.deepdetect.com/) is a machine learning API and server written in C++11. It makes state

JoliBrain 2.4k Jun 30, 2022
Forward - A library for high performance deep learning inference on NVIDIA GPUs

a library for high performance deep learning inference on NVIDIA GPUs.

Tencent 123 Mar 17, 2021
A library for high performance deep learning inference on NVIDIA GPUs.

Forward - A library for high performance deep learning inference on NVIDIA GPUs Forward - A library for high performance deep learning inference on NV

Tencent 502 May 31, 2022
Deploying Deep Learning Models in C++: BERT Language Model

This repository show the code to deploy a deep learning model serialized and running in C++ backend.

null 42 Mar 24, 2022
TFCC is a C++ deep learning inference framework.

TFCC is a C++ deep learning inference framework.

Tencent 108 May 19, 2022
AI4Animation: Deep Learning, Character Animation, Control

This project explores the opportunities of deep learning for character animation and control as part of my Ph.D. research at the University of Edinburgh in the School of Informatics, supervised by Taku Komura. Over the last couple years, this project has become a modular and stable framework for data-driven character animation, including data processing, network training and runtime control, developed in Unity3D / Tensorflow / PyTorch.

Sebastian Starke 5.1k Jun 28, 2022
KSAI Lite is a deep learning inference framework of kingsoft, based on tensorflow lite

KSAI Lite English | 简体中文 KSAI Lite是一个轻量级、灵活性强、高性能且易于扩展的深度学习推理框架,底层基于tensorflow lite,定位支持包括移动端、嵌入式以及服务器端在内的多硬件平台。 当前KSAI Lite已经应用在金山office内部业务中,并逐步支持金山

null 75 Apr 14, 2022
Deep Learning in C Programming Language. Provides an easy way to create and train ANNs.

cDNN is a Deep Learning Library written in C Programming Language. cDNN provides functions that can be used to create Artificial Neural Networks (ANN)

Vishal R 11 Apr 18, 2022
Triton - a language and compiler for writing highly efficient custom Deep-Learning primitives.

Triton - a language and compiler for writing highly efficient custom Deep-Learning primitives.

OpenAI 3.7k Jul 1, 2022
deep learning vision detector/estimator

libopenvision deep learning visualization C library Prerequest ncnn Install openmp vulkan(optional) Build git submodule update --init --recursuve cd b

Prof Syd Xu 2 Feb 8, 2022