Immersive IMM Format and Player

Related tags

Miscellaneous IMM
Overview

Introduction

Immersive media (IMM) is an API-neutral runtime immersive media delivery format. IMM provides an efficient, extensible, interoperable format for the transmission and loading of immersive 3D and 2D animated content of mixed media types (geometry, pictures, 360 panoramas, stroke based paintings, etc).

Context

With the advent of VR/AR technology and platforms, filmic and animated storytelling can happen in 6 degrees of freedom, in that the viewer is located in the same space where the story is being told. Unlike traditional film or 3D animation where the final delivery format is 2D pixels, for VR storytelling the content needs to stay 3D until the very moment it is presented to the user. This requires the equivalent of a new media file format that can handle true immersion. Unlike depth-based 360 stereo video or light-fields based video, IMM is designed to transmit a true, full 3D description of the film.

This is achieved by honoring the original 3D nature of the content. 3D models, 3D paint strokes, voxel data, and other 3D content all have special containers inside IMM so that the playback engine can produce a true immersive rendition. In addition, IMM has containers for more traditional pieces of information such as 2D pictures (positioned in 3D space), audio, 360 backgrounds, etc. IMM comes also with a scenegraph and an animation timeline, so the playback engine can reproduce the film appropriately. All data types are heavily compressed for quick streaming of the data from the internet to the user's device.

The current IMM repository contains the IMM exporter and importer, as well as a reference playback engine.

IMM has been used to deliver a few dozen films, including the Tribeca film festival nominated "Rebels", the "Tale of Soda Island" series, “The Remedy”, "Goodbye Mr. Octopus", "4 Stories" and many more.

IMM Basics

IMM files store binary data. Scenegraph and animation metadata is uncompressed binary for easy streaming and rapid parsing of large files. This usually represents a negligible fraction of a film's file size. On the other hand, all asset data is binary compressed for efficient storage, with a specific compression tailored for each container type. The asset data is readable in random order, and it is recommended the playback engine loads it on demand and streamlined in and out of memory as needed by the scenegraph and animation timeline. The reference player in the IMM repository shows how to do this.

Versioning

IMM is a living format, and it's expected to evolve rapidly together with the VR animation industry. Because of that, each data container comes with a versioning schema that can be used to keep backwards compatibility as needed.

Project architecture

Modules

The IMM project contains a set of libraries, binaries and extra files. You'll find them all at the root of the IMM/ folder:

Name Type Description
libImmExporter/ library Exporting a scene graph into IMM
libImmImporter/ library Reading an IMM file into memory
libImmPlayer/ library Reference player capable of plating IMM files (GL and DX renderers)
libCore/ library OS services, containers, Rendering, Sound, etc
appImmViewer/ binary An native IMM player for Windows, base on libImmPlayer
appImmUnity/ binary A Unity IMM player plugin, based on libImmPlayer
appDX11ShaderCompiler/ binary A command line utility to compile DX11 shader, needed by libImmPlayer
ImmUnitySampleProject/ project A Unity project showing how to use appImmUnity
projects/ project Contains all the Visual Studio and Android project files to build Imm

Dependencies

This is the dependency hierarchy for IMM playback solutions

fig1

This is the dependency hierarchy for IMM import and export pipelines. Note, we do recommend NOT importing IMM files for further art authoring, since IMM has already been optimized for storage, transmission and playback. Think of the IMM as a JPG - if you want to modify its content, you probably want to do it in the source PSD or PNG file and re-export to JPG again.

fig2

Building the Libraries

The ImmViewer works both on Mono and in VR (either with Oculus RIFT or Oculus Quest + Link) or for monoscopic rendering.

On Android (for Quest)

  1. Download Android Studio (version 4.1.0) from developer.android.com.

  2. Open folder thirdparty/lpng1637/projects/androidstudio in Android Studio first and build the project.

  3. Open projects/android/ in android studio and build libCore, libImmImporte, libImmPlayer, appImmViewer in order.

  4. Connect your Quest or Quest 2 Device to your computer and follow the Developer Guide to enable Developer mode on your device. There will be a Quest device showing up on your Android Studio.

  5. Add a new Android App runnable in Configurations. Set the module as android.appImmViewer and click the play button on the menu bar.

You might also like...
 A plugin that can display player information overhead
A plugin that can display player information overhead

A plugin that can display player information overhead Config File At plugins/HeadShow/config.json { "updateTick":60,

(R) Efficient methods and operators for the sparse matrix classes in 'Matrix' (esp. CSR format or "RsparseMatrix")

MatrixExtra MatrixExtra is an R package which extends the sparse matrix and sparse vector types in the Matrix package, particularly the CSR or Rsparse

Standardise code formating for cmake projects with git and clang-format

git-cmake-format This project aims to provide a quick and easy way to integrate clang-format into your CMake project hosted in a git repository, it co

IconVG is a compact, binary format for simple vector graphics: icons, logos, glyphs and emoji.
IconVG is a compact, binary format for simple vector graphics: icons, logos, glyphs and emoji.

IconVG IconVG is a compact, binary format for simple vector graphics: icons, logos, glyphs and emoji. WARNING: THIS FORMAT IS EXPERIMENTAL AND SUBJECT

Converts file formats supported by AdPlug to VGM format.

adlib2vgm Converts file formats supported by AdPlug to VGM format. Notice This tool is originally designed to work with the RetroWave OPL3 Sound Card.

A free, open-source compressor for the ZX0 format

salvador -- a fast, near-optimal compressor for the ZX0 format salvador is a command-line tool and a library that compresses bitstreams in the ZX0 for

format lua code

EmmyLuaCodeStyle 项目介绍 该项目是基于C++的lua代码格式化算法库 经过长期实践,发现人们对格式化算法的预期是尽可能少的改动代码的行布局,而列布局符合基本审美就可以了。 基于这样的想法我设计并实现了lua格式化算法 格式化行为介绍 基本语句 该算法的主要特点是分析并对当前代码做出

A format converter for surface mesh intergrated with muli tools.
A format converter for surface mesh intergrated with muli tools.

MeshConverter A format converter for surface mesh intergrated with small tools for fast surface mesh modification in CFD application scenarios. Build

Quite OK Image (QOI) format encoder/decoder

This project implements encoding and decoding the "Quite OK Image" (QOI) format in the Ć programming language. Ć can be automatically translated to pu

Comments
  • Binaries/builds

    Binaries/builds

    Hi, Do you plan to release binaries? I'm not a C++ dev and the project doesn't build out of the box on Windows. I don't have the expertise or inclination to fix the issues but I'm interested in working on the Unity side of things.

    opened by andybak 9
  • Fix unity plugin(PC)

    Fix unity plugin(PC)

    I went through the entire process including a clean pull and setup from scratch. This pull request fixes all the issues I found along the way. Hopefully these changes make it clear and easier for new users to use.

    opened by alegna901 1
  • ImmUnitySampleProject is broken

    ImmUnitySampleProject is broken

    Several issues:

    1. The class ImmViewer lives in a file called QuillViewer which causes an error
    2. Possibly because of this all script references in the scene are broken
    3. The project is missing the Packages and ProjectSettings directories which can cause many problems down the line
    4. the .gitignore should be based on https://github.com/github/gitignore/blob/master/Unity.gitignore with the Unity paths adjusted to reflect the fact the Unity project isn't the root of the repo. (Actually - I would probably recommend splitting the Unity project out to it's own repo - it usually makes life a lot simpler)
    opened by andybak 1
Owner
null
The OpenEXR project provides the specification and reference implementation of the EXR file format, the professional-grade image storage format of the motion picture industry.

OpenEXR OpenEXR provides the specification and reference implementation of the EXR file format, the professional-grade image storage format of the mot

Academy Software Foundation 1.3k Sep 26, 2022
(Simple String Format) is an syntax of format and a library for parse this.

SSFMT (Simple String Format) is an syntax of format and a library for parse this. SSFMT != {fmt} SSFMT is NOT an API/library for parse {fmt} syntax !

null 2 Jan 30, 2022
Cross-platform tool to extract wavetables and draw envelopes from sample files, exporting the wavetable and generating the appropriate SFZ text to use in a suitable player.

wextract Cross-platform tool to extract wavetables and draw envelopes from sample files, exporting the wavetable and generating the appropriate SFZ te

Paul Ferrand 9 Jan 5, 2022
📽 A simple X11+SDL2 animated wallpaper setter and video player

anipaper ?? A simple X11+SDL2 animated wallpaper setter and video player Introduction Anipaper (ANImated Wallpaper) is a simple 'wallpaper setter' for

Davidson Francis 8 Apr 21, 2022
Quake Enhanced mod where one player (The Juggernaut) is very strong and all other players have to kill the Juggernaut

QE Juggernaut Quake Enhanced Juggernaut (A modification of the QEHunter mod by JPiolho.) This is a multiplayer mod where one player is the Juggernaut.

null 2 Jun 6, 2022
Simple Player-Glow & Driver Source Included

External-Apex-Cheat Install WDK Build in Release x64 Map driver using KDMapper Launch Game Run User-mode when in main menu Have Fun driver is indeed p

null 97 Sep 13, 2022
ESP32 drum computer / sample player / midi sequencer (Arduino audio project)

esp32_drum_computer ESP32 drum computer / sample player / midi sequencer (Arduino audio project) The project can be seen in my video https://youtu.be/

Marcel 36 Sep 4, 2022
GTA Online survival missions in Single Player mode

SurvivalsModCPPVersion You liked the survival missions from GTA Online? Well, me too. This mod aims to recreate the same gamemode in Single Player mod

null 1 Nov 23, 2021
PUBG ESP Hack for Emulator using C++ code. Player Position, Bones, Loots, Weapons, Vehicles, Boxes ... etc.

PUBG 1.7 ESP Hack for Emulator (C++ Source Code) PUBG ESP Hack for Emulator using C++ code. Player Position, Bones, Loots, Weapons, Vehicles, Boxes ..

Zero One Billion 33 Sep 24, 2022
A small XM (FastTracker II Extended Module) player library.

libxm A small XM (FastTracker II Extended Module) player library. Main features: Small size in mind; many features can be disabled at compile-time, or

Romain D. 104 Sep 18, 2022