A method to estimate 3D hand gestures from 3D body motion input in body languages.

Overview

Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics (CVPR 2021)

report

Project page

This repository contains a pytorch implementation of "Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics"

SOTA vs. Ours:

This codebase provides:

  • train code
  • test code
  • visualization code
  • plug-in for smplx test code

Overview:

We provide models and code to train/test the body only prior model and the body + image model

Body only vs. Body + image as input:

See below for sections:

  • Installation: environment setup and installation for visualization
  • Download data and models: download annotations and pre-trained models
  • Training from scratch: scripts to get the training pipeline running from scratch
  • Testing with pretrianed models: scripts to test pretrained models and to visualize outputs
  • SMPLx Plug-in Demo: scripts to run models on SMPLx body model outputs from FrankMocap

Installation:

with Cuda version 10.1

conda create -n venv_b2h python=3.7
conda activate venv_b2h
pip install -r requirements.txt

Visualization (Optional):

Please follow the installation instructions outlined in the MTC repo.

Download data and models:

Downloading data described here.

Training from scratch:

## Flag definitions:
## --models: path to directory save checkpoints
## --base_path: path to directory where the directory video_data/ is stored
## --require_image: whether or not to include resnet feature as input

## to train with body only as input
python train_gan.py --model models/ --base_path ./`

## to train with body and image as input
python train_gan.py --model models/ --base_path ./` --require_image

Testing with pretrained models:

To test with provided pretrained models (see above section "Download"). If training from scratch, replace --checkpoint as necessary.

## Flag definitions:
## --checkpoint: path to saved pretrained model
## --data_dir: directory of the test data where the .npy files are saved
## --base_path: path to directory where the directory video_data/ is stored
## --require_image: whether or not to include resnet feature as input
## --tag: (optional) naming prefix for saving results

## testing model with body only as input
python sample.py --checkpoint models/ours_wb_arm2wh_checkpoint.pth \
                 --data_dir video_data/Multi/sample/ \
                 --base_path ./ \
                 --tag 'test_'


## testing model with body and image as input
python sample.py --checkpoint models/ours_wbi_arm2wh_checkpoint.pth \
                 --data_dir video_data/Multi/sample/ \
                 --base_path ./ \
                 --tag 'test_wim_' \
                 --require_image

After running the above code, you can check to see if your outputs match the provided outputs we provide under video_data/Multi/sample/chemistry_test/seq1/sample_results/test_predicted_body_3d_frontal/<%04d.txt> or video_data/Multi/sample/chemistry_test/seq1/sample_results/test_wim_predicted_body_3d_frontal/<%04d.txt> depending on if you run with body only input or body+image input respectively.

Visualization (Optional)

Once you have run the above test script, the output will be saved to a <path_to_sequence>/results/<tag>_predicted_body_3d_frontal/ directory as .txt files for each frame. We can then visualize the results as follows (see MTC repo for installation instructions):

## to visualize results from above test commands, run ./wrapper.sh <sequence path> <tag>

cd visualization/

## visualize model with body only as input
./wrapper.sh ../video_data/Multi/sample/chemistry_test/seq1/ test_

## visualize model with body and image as input
./wrapper.sh ../video_data/Multi/sample/chemistry_test/seq1/ test_wim_

After running the above visualization code, you can check to see if the first few generated visualizations match ours by checking video_data/Multi/sample/chemistry_test/seq1/sample_results/test_predicted_body_3d_frontal/<%04d.png> or video_data/Multi/sample/chemistry_test/seq1/sample_results/test_wim_predicted_body_3d_frontal/<%04d.png> depending on if you run with body only input or body+image input respectively.

SMPLx Plug-in Demo

We also provide a quick plug-in script for SMPLx body model compatibility using FrankMocap to obtain the 3D body poses (as opposed to Adam body model using MTC). Please refer to Frankmocap repo for installation instructions. Once the package is properly installed, follow the instructions python -m demo.demo_frankmocap --input_path <path_to_mp4> --out_dir <path_to_output> to generate files: <path_to_output>_prediction_result.pkl.

For the purposes of this demo, we provide a short example of expected FrankMocap outputs under video_data/Multi/conan_frank/mocap/

## Flag definitions:
## -- checkpoint: path to saved pretrained model
## --data_dir:  directory where all of the `*_prediction_result.pkl` files are saved
## --tag (optional) naming prefix for saving results

## to run on output smplx files from frankmocap
python -m smplx_plugin.demo --checkpoint models/ours_wb_arm2wh_checkpoint.pth \
                            --data_dir video_data/Multi/conan_frank/mocap/ \
                            --tag 'mocap_'

## to visualize
cd visualization && ./wrapper.sh ../video_data/Multi/conan_frank/mocap/ mocap_

Coming Soon!

  • Pretrained models (Ours w/ B and Ours w/ B+I)
  • Saved results from our methods as additional downloadable data

License

Comments
  • Error when loading smplx model

    Error when loading smplx model

    Hi, thanks for the fantastic work!

    When I tried the SMPLx Plug-in Demo with the command python -m smplx_plugin.demo --checkpoint models/ours_wb_arm2wh_checkpoint.pth --data_dir video_data/Multi/conan_frank/mocap/ --tag 'mocap_', I met this error:

    Traceback (most recent call last): File "/home/sunyn/.pycharm_helpers/pydev/pydevd.py", line 1483, in _exec pydev_imports.execfile(file, globals, locals) # execute the script File "/home/sunyn/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "/data/sunyn/MotionInBetween/body2hands/smplx_plugin/demo.py", line 166, in main(args) File "/data/sunyn/MotionInBetween/body2hands/smplx_plugin/demo.py", line 50, in main test_X, total_body, total_cam = load_smplx(args.data_dir) File "/data/sunyn/MotionInBetween/body2hands/smplx_plugin/demo.py", line 142, in load_smplx body = aa_to_rot6d(body) File "/data/sunyn/MotionInBetween/body2hands/utils/load_utils.py", line 100, in aa_to_rot6d np_mat = np_mat.as_dcm() AttributeError: 'scipy.spatial.transform.rotation.Rotation' object has no attribute 'as_dcm'

    I have not changed any code and already downloaded all the required files. Besides, the Testing with pretrained models works well. I have no idea how to fix this bug. Could you please give me some advice? Thanks a lot.

    opened by YananSunn 1
  • The pretrained Model

    The pretrained Model

    Hello. Thanks for the code. I'm having trouble training because of my video card limitations (hardware) :(. Is there an approximate release date for the pre-trained model?

    opened by italosalgado14 1
  • Dimension of full_bodies2.npy doesn't match description

    Dimension of full_bodies2.npy doesn't match description

    Hi,

    I found that the downloaded full_bodies2.npy has dimension of (N,T,120), which is different from the description for downloading, where it says it should be (N,T,36). Then I noticed that in your data preprocessing code, ARMS_ONLY joints are being selected to 6 joints thus the dimension becomes 36.

    p0_windows = np.reshape(p0_windows, (B,T,-1,6))
    p0_windows = p0_windows[:,:,ARMS_ONLY,:]
    p0_windows = np.reshape(p0_windows, (B,T,-1))
    

    I assume that loading from npy file and indexing using ARMS_ONLY should work. However, I'm not sure if this procedure is correct due to the mismatch between the downloaded file and your description. Could you provide some details regarding this? Thanks in advance.

    Another question is that, is full_bodies2.npy corresponding to MTC's output? Should I look for reference in MTC repo in order to understand which joint the value in full_bodies2.npy and full_hands2.npy corresponds to? Looking forward to your reply!

    opened by wubowen416 0
  • Create own test data

    Create own test data

    Hi, thanks for the fantastic work.

    I'm trying to use my own data to test and have read the [using-your-own-data-for-train] in readme, but still confused about how to generate full_resnet.npy. Could you please explain it with more details? (Maybe with some codes if necessary.) Thanks a lot!

    opened by YananSunn 0
  • fix print statements

    fix print statements

    In lines 120, 122, and 137, there are print statements lacking parentheses (), which is incorrect in Python 3 and causes the execution to crash. I've rewritten these print statements to their correct form.

    CLA Signed 
    opened by alvaro-budria 0
  • Evaluation code

    Evaluation code

    Can you offer the evaluation code, which can report average error over the entire sequence, and scale the error unit to millimeters based on a reference shoulder distance of 30 cm?

    opened by Fengweinlpr 0
Owner
Facebook Research
Facebook Research
Convert ASF/AMC motion capture files to BVH motion capture files.

About amc2bvh is a utility that converts a pair of files, one in the Acclaim Skeleton Format (ASF) and the other in the Acclaim Motion Capture (AMC) f

Tom Copeland 9 Nov 10, 2022
Python Plug-in for the Rime Input Method Engine

librime-python 为什么你需要 librime-python? 1. Python 是解释执行的脚本语言,因此可以在 rime 输入法的基础上动态地添加功能,而无需重新编译源代码、重启输入法程序,甚至无需重新部署。 2. 与 librime 本身使用的 C++ 语言相比,Python 提

Ayaka Mikazuki 15 Sep 4, 2022
This is a small example project, that showcases the possibility of using a surrogate model to estimate the drag coefficient of arbitrary triangles.

flowAroundTriangles This is a small example project, that showcases the possibility of using a surrogate model to estimate the drag coefficient of arb

null 6 Sep 16, 2022
HugeCTR is a GPU-accelerated recommender framework designed to distribute training across multiple GPUs and nodes and estimate Click-Through Rates (CTRs)

Merlin: HugeCTR HugeCTR is a GPU-accelerated recommender framework designed to distribute training across multiple GPUs and nodes and estimate Click-T

null 745 Dec 1, 2022
Android hand detect and pose estimation by ncnn

ncnn_nanodet_hand 1.hand detect:用nanode-m训练了个hand detect模型, 2.hand pose:用CMU的数据集训练了个ghostnet作为backbone模仿pfld的handpose模型 3.推理:handpose.cpp单独检测pose,nano

null 78 Nov 18, 2022
Android yolox hand detect by ncnn

The yolox hand detection This is a sample ncnn android project, it depends on ncnn library and opencv https://github.com/Tencent/ncnn https://github.c

FeiGeChuanShu 14 Sep 7, 2022
Real-Time Neural 3D Hand Pose Estimation from an Event Stream [ICCV 2021]

EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream Project Page Index TRAIN.md -- how to train the model from scratch EVAL_REAL

null 23 Nov 7, 2022
ManipNet: Neural Manipulation Synthesis with a Hand-Object Spatial Representation - SIGGRAPH 2021

ManipNet: Neural Manipulation Synthesis with a Hand-Object Spatial Representation - SIGGRAPH 2021 Dataset Code Demos Authors: He Zhang, Yuting Ye, Tak

HE ZHANG 195 Dec 1, 2022
A mediapipe-hand demo infer by ncnn

The Android demo of Mediapipe Hand infer by ncnn Please enjoy the mediapipe hand demo on ncnn You can try this APK demo https://pan.baidu.com/s/1ArAMH

FeiGeChuanShu 47 Nov 29, 2022
Apex Legends Cheat Season 8 With DIRECT EFI Method By TheCruz

CRZAIMBOT How To Use Just Compile The Source with Release Option Find CRZAimbot.dll on x64\release\ folder Make Sure you already load memory.efi Open

Rasyid Abdul Halim S.Tr.kom 33 May 27, 2022
OpenPose: Real-time multi-person keypoint detection library for body, face, hands, and foot estimation

Build Type Linux MacOS Windows Build Status OpenPose has represented the first real-time multi-person system to jointly detect human body, hand, facia

null 25.4k Nov 27, 2022
JBDL: A JAX-Based Body Dynamics Algorithm Library forRobotics

JBDL: A JAX-Based Body Dynamics Algorithm Library forRobotics

Tencent Robotics X 20 Nov 9, 2022
Extracts high-precision mouse/pointer motion data on Windows. Good for drawing software!

window_mouse_queue This is a wrapper for GetMouseMovePointsEx function that allows to extract high-precision mouse/pointer motion data on Windows. Goo

YellowAfterlife's GameMaker Things 6 Feb 21, 2022
Stochastic Scene-Aware Motion Prediction

Stochastic Scene-Aware Motion Prediction [Project Page] [Paper] Description This repository contains the runtime Unity code for SAMP. For the training

Mohamed Hassan 97 Nov 9, 2022
ROS wrapper for real-time incremental event-based vision motion estimation by dispersion minimisation

event_emin_ros ROS wrapper for real-time incremental event-based vision motion estimation by dispersion minimisation (EventEMin). This code was used t

Imperial College London 2 Jan 10, 2022
open Multiple View Geometry library. Basis for 3D computer vision and Structure from Motion.

OpenMVG (open Multiple View Geometry) License Documentation Continuous Integration (Linux/MacOs/Windows) Build Code Quality Chat Wiki local/docker bui

openMVG 4.5k Nov 25, 2022
A tiny C++11 library for reading BVH motion capture data

bvh11 A tiny C++11 library for reading (and writing) BVH motion capture data. Dependencies C++11 standard library Eigen 3 http://eigen.tuxfamily.org/

Yuki Koyama 33 Nov 17, 2022
Pixel-Perfect Structure-from-Motion with Featuremetric Refinement (ICCV 2021, Oral)

Pixel-Perfect Structure-from-Motion Best student paper award @ ICCV 2021 We introduce a framework that improves the accuracy of Structure-from-Motion

Computer Vision and Geometry Lab 792 Nov 24, 2022