UE4 MediaPipe plugin

Overview

UE4 MediaPipe plugin

Platforms: Win64

2D features: Face, Iris, Hands, Pose, Holistic

3D features: Face Mesh, World Pose

Demo video: https://www.youtube.com/watch?v=_gRGjGn6FQE

More details: https://github.com/google/mediapipe

Custom build howto

Clone plugin

git clone https://github.com/wongfei/ue4-mediapipe-plugin.git

Clone wrapper

git clone -b unreal https://github.com/wongfei/mediapipe.git ue4-mediapipe-wrapper

Setup workspace

Build wrapper

cd ue4-mediapipe-wrapper

bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 --action_env PYTHON_BIN_PATH="C:\\Python39\\python.exe" mediapipe/unreal:ump_shared

More info: ue4-mediapipe-wrapper\mediapipe\unreal\scripts\build_shared.cmd

Copy ump_shared.dll to ue4-mediapipe-plugin\Plugins\MediaPipe\ThirdParty\mediapipe\Binaries\Win64\

More info: ue4-mediapipe-wrapper\mediapipe\unreal\scripts\deploy.cmd

Other deps

Protobuf 3.11.4

Copy libprotobuf.lib to ue4-mediapipe-plugin\Plugins\MediaPipe\ThirdParty\protobuf\Lib\Win64\

OpenCV 3.4.10 with fixed CAP_MSMF (optional)

git clone -b msmf_fix_3410 https://github.com/wongfei/opencv.git

Copy opencv_world3410.dll to ue4-mediapipe-plugin\Plugins\MediaPipe\ThirdParty\mediapipe\Binaries\Win64\

Comments
  • Some problems in practical use

    Some problems in practical use

    Visibility and presence should mean the same thing. I don't understand the difference between the two image Pose point has only position vector and no directional Euler angle, which leads to low efficiency in driving the model image I try to calculate by mathematical operation, but the effect is not very ideal. image Since last year, I have had the idea of porting mediapipe to fantasy. But my code is very bad. I spend more time using blueprints. Thank you very much for making this plug-in. I want to make it better

    opened by salier 24
  • The project does not open in UE 5.0.2

    The project does not open in UE 5.0.2

    If I try to open a project in UE 5.0.2, it will stop at 75% Initializing.

    When I checked the log, I checked that there was no response after opening the ump_shared.dll file for the last time, but I'm not sure if this dll is really the problem.

    How can I fix this?

    Thank you for reading it.

    opened by hyunwoolee0099 10
  • MacOS and HTML5

    MacOS and HTML5

    Hi Wongfei,

    I am currently looking into the code an dusage of the dlls and protobuf.

    I am sure with some work this plugin could be edited to also include the MacOS platform, right?

    And would it work to build to an HTML5 package?

    Are you available for commission?

    Thanks,

    Aaron

    Edit: I asked my colleague on Windows to open the project but he got the same error of not being able to build. He was on UE4.26.2.

    opened by aaronvdbrugge 7
  •  LogMediaPipe: Error: ObjectId out of range

    LogMediaPipe: Error: ObjectId out of range

    log.txt An error is reported at runtime logmediapipe: error: objectid out of range The number of frames is greatly reduced (from 200 frames to about 60 frames) The array length of 0 is the camera start-up time (or the face is not recognized)

    opened by salier 7
  • Video Feed before MediaPipe processing

    Video Feed before MediaPipe processing

    Hey Wei Fong,

    Is there a setting to turn off the rendering of the Landmarks in the video texture? I'd like to still use the Video without the landmark rendering. Or maybe is there somewhere I can use the video feed before it gets processed by mediapipe?

    opened by tbKorber 5
  • Turning the 'enable overlay' variable to false breaks the hand landmarks.

    Turning the 'enable overlay' variable to false breaks the hand landmarks.

    Hello! I am nearing my project's end using this awesome plugin, but I'm facing a slight issue wrapping things up.

    I don't want the Mediapipe window showing the raw camera feed to pop up on the final packaged project. Turning the 'enable overlay' variable to false seems to do that, but it greatly reduces the performance of the hand landmark tracking.

    My intended outcome is simply to prevent the extra window showing the webcam feed from popping up. Am I doing this the wrong way?

    Thank you.

    opened by EDsizm 4
  • UE503 - Stuck at 75% after compiling while using Unreal 5.0.3

    UE503 - Stuck at 75% after compiling while using Unreal 5.0.3

    The plugin seems compiling correctly, but remains stuck at 75% while loading the demo project.

    image

    Recompile through VS19 going positive:

    Ricompilazione avviata... 1>------ Inizio ricompilazione: Progetto: MediaPipeDemo, Configurazione: Development_Editor x64 ------ 1>Cleaning MediaPipeDemoEditor and UnrealHeaderTool binaries... 1>Using bundled DotNet SDK 1>Log file: C:\Users\Francesco\AppData\Local\UnrealBuildTool\Log.txt 1>Creating makefile for MediaPipeDemoEditor (no existing makefile) 1>Parsing headers for MediaPipeDemoEditor 1> Running UnrealHeaderTool "C:\Users\Francesco\Desktop\ue4-mediapipe-plugin-master\MediaPipeDemo.uproject" "C:\Users\Francesco\Desktop\ue4-mediapipe-plugin-master\Intermediate\Build\Win64\MediaPipeDemoEditor\Development\MediaPipeDemoEditor.uhtmanifest" -LogCmds="loginit warning, logexit warning, logdatabase error" -Unattended -WarningsAsErrors -abslog="C:\Users\Francesco\AppData\Local\UnrealBuildTool\Log_UHT.txt" -installed 1>Reflection code generated for MediaPipeDemoEditor in 4,5386259 seconds 1>Building MediaPipeDemoEditor... 1>Using Visual Studio 2019 14.29.30145 toolchain (C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133) and Windows 10.0.18362.0 SDK (C:\Program Files (x86)\Windows Kits\10). 1>[Adaptive Build] Excluded from MediaPipe unity file: DynamicTexture.cpp, MediaPipeClassificationObserverComponent.cpp, MediaPipeFaceMeshObserverComponent.cpp, MediaPipeLandmarkObserverComponent.cpp, MediaPipeModule.cpp, MediaPipeObserverComponent.cpp, MediaPipePipelineComponent.cpp, MediaPipePresenceObserverComponent.cpp 1>[Adaptive Build] Excluded from MediaPipeDemo unity file: MediaPipeCppExample.cpp, MediaPipeDemo.cpp 1>Determining max actions to execute in parallel (4 physical cores, 8 logical cores) 1> Executing up to 4 processes, one per physical core 1>Building 44 actions with 4 processes... 1>[1/44] Resource Default.rc2 1>[2/44] Compile optical_flow_field_data.pb.cc 1>[3/44] Compile time_series_header.pb.cc 1>[4/44] Compile environment.pb.cc 1>[5/44] Compile MediaPipe.init.gen.cpp 1>[6/44] Compile anchor.pb.cc 1>[7/44] Compile mesh_3d.pb.cc 1>[8/44] Compile image_file_properties.pb.cc 1>[9/44] Compile MediaPipeClassificationObserverComponent.gen.cpp 1>[10/44] Compile image_format.pb.cc 1>[11/44] Compile landmark.pb.cc 1>[12/44] Compile rect.pb.cc 1>[13/44] Compile rasterization.pb.cc 1>[14/44] Compile MediaPipeLandmarkObserverComponent.gen.cpp 1>[15/44] Compile locus.pb.cc 1>[16/44] Compile MediaPipePresenceObserverComponent.cpp 1>[17/44] Compile matrix_data.pb.cc 1>[18/44] Compile MediaPipeObserverComponent.cpp 1>[19/44] Compile detection.pb.cc 1>[20/44] Compile MediaPipeFaceMeshObserverComponent.cpp 1>[21/44] Compile MediaPipeObserverComponent.gen.cpp 1>[22/44] Compile classification.pb.cc 1>[23/44] Compile SharedPCH.Engine.ShadowErrors.cpp 1>[24/44] Compile MediaPipePipelineComponent.gen.cpp 1>[25/44] Compile geometry_pipeline_metadata.pb.cc 1>[26/44] Compile MediaPipePresenceObserverComponent.gen.cpp 1>[27/44] Compile DynamicTexture.cpp 1>[28/44] Compile location_data.pb.cc 1>[29/44] Compile MediaPipeModule.cpp 1>[30/44] Compile MediaPipePipelineComponent.cpp 1>[31/44] Compile face_geometry.pb.cc 1>[32/44] Compile MediaPipeDemo.init.gen.cpp 1>[33/44] Compile MediaPipeFaceMeshObserverComponent.gen.cpp 1>[34/44] Resource Default.rc2 1>[35/44] Compile MediaPipeDemo.cpp 1>[36/44] Compile MediaPipeCppExample.gen.cpp 1>[37/44] Compile MediaPipeCppExample.cpp 1>[38/44] Link UnrealEditor-MediaPipeDemo.lib 1> Creazione della libreria C:\Users\Francesco\Desktop\ue4-mediapipe-plugin-master\Intermediate\Build\Win64\UnrealEditor\Development\MediaPipeDemo\UnrealEditor-MediaPipeDemo.lib e dell'oggetto C:\Users\Francesco\Desktop\ue4-mediapipe-plugin-master\Intermediate\Build\Win64\UnrealEditor\Development\MediaPipeDemo\UnrealEditor-MediaPipeDemo.exp 1>[39/44] Compile MediaPipeLandmarkObserverComponent.cpp 1>[40/44] Compile MediaPipeClassificationObserverComponent.cpp 1>[41/44] Link UnrealEditor-MediaPipe.lib 1> Creazione della libreria C:\Users\Francesco\Desktop\ue4-mediapipe-plugin-master\Plugins\MediaPipe\Intermediate\Build\Win64\UnrealEditor\Development\MediaPipe\UnrealEditor-MediaPipe.lib e dell'oggetto C:\Users\Francesco\Desktop\ue4-mediapipe-plugin-master\Plugins\MediaPipe\Intermediate\Build\Win64\UnrealEditor\Development\MediaPipe\UnrealEditor-MediaPipe.exp 1>[42/44] Link UnrealEditor-MediaPipe.dll 1> Creazione della libreria C:\Users\Francesco\Desktop\ue4-mediapipe-plugin-master\Plugins\MediaPipe\Intermediate\Build\Win64\UnrealEditor\Development\MediaPipe\UnrealEditor-MediaPipe.suppressed.lib e dell'oggetto C:\Users\Francesco\Desktop\ue4-mediapipe-plugin-master\Plugins\MediaPipe\Intermediate\Build\Win64\UnrealEditor\Development\MediaPipe\UnrealEditor-MediaPipe.suppressed.exp 1>[43/44] Link UnrealEditor-MediaPipeDemo.dll 1> Creazione della libreria C:\Users\Francesco\Desktop\ue4-mediapipe-plugin-master\Intermediate\Build\Win64\UnrealEditor\Development\MediaPipeDemo\UnrealEditor-MediaPipeDemo.suppressed.lib e dell'oggetto C:\Users\Francesco\Desktop\ue4-mediapipe-plugin-master\Intermediate\Build\Win64\UnrealEditor\Development\MediaPipeDemo\UnrealEditor-MediaPipeDemo.suppressed.exp 1>[44/44] WriteMetadata MediaPipeDemoEditor.target 1>Total time in Parallel executor: 139,31 seconds 1>Total execution time: 149,50 seconds ========== Ricompilazione: 1 completate, 0 non riuscite, 0 ignorate ==========

    opened by fpibbs 3
  • Questions about code location

    Questions about code location

    Questions where can I check the code that connects the landmark of the media pipe to the skeleton bones of the unreal engine? I want to learn that process! thanks

    opened by donguk071 3
  • BlackScreen after Package for windows. UE 5.01

    BlackScreen after Package for windows. UE 5.01

    Good afternoon. The last source. When packing a project under Windows, I can't start the resulting application. The application window freezes and nothing else happens. Even on an empty map. In the editor, everything starts and works like a charm.

    opened by UmkaApps 3
  • holistic landmarks detection does not output hand detection confidence

    holistic landmarks detection does not output hand detection confidence

    Hi! Wongfei, When running with this plugin, I encountered the following problem: The hand landmarks detection (configured by multi_hand_landmarks.pbtxt) output hand detection confidence, while the holistic landmarks detection (configured by holistic_landmarks.pbtxt or holistic_with_iris.pbtxt) does not output the hand detection confidence. How to solve this issue? Can I just modify holistic_landmarks.pbtxt to add the corresponding output without changing the code structure. Looking forward to your reply!

    opened by increase24 3
  • Compiled the UE4 project. How to connect the camera?

    Compiled the UE4 project. How to connect the camera?

    I have compiled the project and when I started to run the level the MediaPipe window appears black. I guess that is not connected to the webcam. Can you explain me where or in which config file can I address the webcam?

    Best regards, Screenshot 2021-07-28 232117

    Mirko

    opened by PinkMediaGroup 3
Releases(v4.26-20211110)
Owner
null
A UE4 editor plugin for better artist/designer UI.

Details Workspace (for UE4) Details Workspace is a UE4 editor plugin for better artist/designer workflow. It allows you to create any number of "Detai

null 4 Apr 29, 2022
MediaPipe offers cross-platform, customizable ML solutions for live and streaming media.

Cross-platform, customizable ML solutions for live and streaming media.

Google 19.5k Nov 27, 2022
Example Qt application that demonstrates how to integrate Mediapipe

Mediapipe integration to Qt application example Example on how to integrate mediapipe as a dynamic library into Qt applicaton on Linux. Resulting appl

null 39 Nov 26, 2022
A mediapipe-hand demo infer by ncnn

The Android demo of Mediapipe Hand infer by ncnn Please enjoy the mediapipe hand demo on ncnn You can try this APK demo https://pan.baidu.com/s/1ArAMH

FeiGeChuanShu 47 Nov 29, 2022
The purpose of this project is to apply mediapipe to more AI chips.

1.About This Project Our Official Website: www.houmo.ai Who We Are: We are Houmo - A Great AI Company. We wish to change the world with unlimited comp

null 38 Nov 20, 2022
Native runtime package for MediaPipe.NET.

MediaPipe.NET.Runtime Native library package for MediaPipe.NET. This is the first half of the port of MediaPipeUnityPlugin, in order to use MediaPipe

Vignette 14 Oct 12, 2022
🦴 Takes base Rokoko animation export and uses my pre-made bone remapping to map to the default UE4 skeleton.

?? RokokoToUE4-pipeline ?? Takes base rokoko animation export and uses my pre-made bone remapping to map to the default UE4 skeleton. How to use: Down

Matthew Jigalin 2 Sep 14, 2022
A light and fast internet speed plugin(DDE).

lfxNet English | 简体中文 | 繁體中文 lfxNet 是一款轻量、快速的实时显示系统资源信息的应用程序。 目录 背景 编译 下载 作者 鸣谢 协议 背景 喜爱 DDE ,为 Deepin 爱好者、也是开发者之一。因习惯其它系统上有一个任务栏网速插件,但在 Deepin/UOS上没有

偕臧 55 Nov 20, 2022
Face tracking plugin for OBS Studio

OBS Face Tracker Plugin Introduction This plugin provide a filter for face detection and face tracking for mainly a speaking person. This plugin emplo

Norihiro Kamae 191 Nov 27, 2022
Gstreamer plugin that allows use of NVIDIA Maxine SDK in a generic pipeline.

GST-NVMAXINE Gstreamer plugin that allows use of NVIDIA MaxineTM sdk in a generic pipeline. This plugin is intended for use with NVIDIA hardware. Visi

Alex Pitrolo 15 Nov 20, 2022
VST/AU Plugin for Auditioning RAVE Models in Real-time

RAVE-audition VST/AU Plugin for Auditioning RAVE Models in Real-time Check out Antoine's great work in the original RAVE repository! Building the C++

Andrew Fyfe 63 Nov 8, 2022
A Qt platform plugin for running Qt/OpenGL applications offscreen

A Qt platform plugin for running Qt/OpenGL applications offscreen https://doc.qt.io/qt-6/qpa.html https://www.khronos.org/registry/EGL/extensions/MESA

null 4 Aug 25, 2022
Insomniac games cache simulation tool plugin for UE4

Insomniac Games CacheSim plugin for UE4 This plugin for Unreal Engine 4 lets you use the Insomniac Games Cache Simulation tool to detect cache misses

Toni Rebollo Berná 29 Aug 27, 2022
UE4 Plugin to execute trained PyTorch modules

SimplePyTorch UE4 Plugin to execute trained PyTorch modules ------- Packaging ------- Download PyTorch C++ distributions: https://pytorch.org/cppdocs/

null 48 Nov 1, 2022
Footsteps Plugin for UE4.

Footsteps Plugin for UE4 Packaged for UE4.27. Example Project See Footsteps Component Propertis if you dont use skeletal mesh or you dont want to use

null 3 Oct 6, 2021
A UE4 plugin used ffmpeg library to record/publish game viewport output and in-game audio.

Streaming A UE4 plugin used ffmpeg library to record/publish game viewport output and in-game audio. GPL license beacuse used ffmpeg. Structures GameV

Luo Yu 3 Aug 3, 2022
A UE4 editor plugin for better artist/designer UI.

Details Workspace (for UE4) Details Workspace is a UE4 editor plugin for better artist/designer workflow. It allows you to create any number of "Detai

null 4 Apr 29, 2022
A UE4 plugin wrapper for Molecular Matter's Live++ Hot-Reloading Library

Good News Everyone: Natively integrated in UE4.22+ Details here: https://docs.unrealengine.com/en-us/Builds/4_22 UE4 LivePP: C/C++ live coding A UE4 p

Kite & Lightning 102 Sep 26, 2022
MediaPipe offers cross-platform, customizable ML solutions for live and streaming media.

Cross-platform, customizable ML solutions for live and streaming media.

Google 19.5k Nov 27, 2022
Example Qt application that demonstrates how to integrate Mediapipe

Mediapipe integration to Qt application example Example on how to integrate mediapipe as a dynamic library into Qt applicaton on Linux. Resulting appl

null 39 Nov 26, 2022