fast MP4 mux / demux using WASM

Related tags

Multimedia mp4-wasm
Overview

mp4-wasm

This module is still under development and may change.

Fast MP4 mux / demux using WASM, for modern browsers and Node.js.

What's supported:

  • MP4 video muxing (taking already-encoded H264 frames and wrapping them in a MP4 container)
  • MP4/H264 encoding and muxing via WebCodecs

What's still WIP:

  • MP4 video demuxing
  • MP4 audio muxing (single AAC track)
  • WebCodecs video decoding and demuxing

This is built on top of the C/C++ library minimp4, and the primary motivator behind this project is to create a hassle-free solution for creating MP4/H264 videos fully client-side in the browser, without running into H264 patent issues.


Docs are WIP.

License

MIT, see LICENSE.md for details.

Comments
  • Using canvas in Nodejs

    Using canvas in Nodejs

    I'm trying to use node canvas to mix the mp4. My approach was to use the canvas.toBuffer method, but it does not seem to work (the output mp4 is a bad file)

    const loadMP4Module = require("../build/mp4.node");
    const fs = require("fs");
    const path = require("path");
    const { promisify } = require("util");
    const readFile = promisify(fs.readFile);
    
    const width = 1920;
    const height = 1080;
    
    const { createCanvas, loadImage } = require('canvas')
    
    
    var MP4;
    var stream;
    
    const drawFrame = (interpolant) => {
        ctx.fillStyle = "#0000FF";
        ctx.fillRect(0, 0, canvas.width, canvas.height);
        ctx.fillStyle = "#FF0000";
        ctx.fillRect(0, 0, canvas.width * interpolant, canvas.height * interpolant);
    };
    
    function write(pointer, size, offset) {
        const buf = MP4.HEAPU8.slice(pointer, pointer + size);
    
        // Write the MP4-muxed chunk directly into a .mp4 stream
        stream.write(buf);
    
        return 0;
    }
    // For an arbitrary stream of data, yield/iterate
    // on each AnnexB NAL chunk (including the startcode)
    function* readNAL(buffer, offset = 0) {
        let h264Size = buffer.byteLength;
        while (h264Size > 0) {
            const nal_size = getNALSize(buffer, offset, h264Size);
            if (nal_size < 4) {
                offset += 1;
                h264Size -= 1;
                continue;
            }
            yield buffer.subarray(offset, offset + nal_size);
            offset += nal_size;
            h264Size -= nal_size;
        }
    
        function getNALSize(buf, ptr, size) {
            let pos = 3;
            while (size - pos > 3) {
                if (
                    buf[ptr + pos] == 0 &&
                    buf[ptr + pos + 1] == 0 &&
                    buf[ptr + pos + 2] == 1
                )
                    return pos;
                if (
                    buf[ptr + pos] == 0 &&
                    buf[ptr + pos + 1] == 0 &&
                    buf[ptr + pos + 2] == 0 &&
                    buf[ptr + pos + 3] == 1
                )
                    return pos;
                pos++;
            }
            return size;
        }
    }
    
    (async () => {
    
        const canvas = createCanvas(width, height)
        const ctx = canvas.getContext('2d')
        // Write "Awesome!"
        ctx.font = '30px Impact'
        ctx.rotate(0.1)
        ctx.fillText('Awesome!', 50, 100)
    
        // Draw line under text
        var text = ctx.measureText('Awesome!')
        ctx.strokeStyle = 'rgba(0,0,0,0.5)'
        ctx.beginPath()
        ctx.lineTo(50, 102)
        ctx.lineTo(50 + text.width, 102)
        ctx.stroke()
    
        stream = fs.createWriteStream(
            path.resolve(__dirname, "outputs/composition.mp4")
        );
    
        const fps = 60;
        const duration = 4;
        let frame = 0;
        let totalFrames = Math.round(fps * duration);
    
        console.time("encode");
    
        MP4 = await loadMP4Module();
        const mux = MP4.create_muxer(
            {
                width,
                height,
                // Needed to write sequentially to a file
                // i.e. no 'offset' parameter needed
                sequential: true,
            },
            write
        );
    
        const file = path.resolve(__dirname, "fixtures/composition");
        // JPEG-encoded, 50% quality
        const buffer = canvas.toBuffer('image/jpeg', { quality: 0.5 })
    
        
        for (let chunk of readNAL(buffer)) {
            // malloc() / free() a pointer
            const p = MP4.create_buffer(chunk.byteLength);
            // set data in memory
            MP4.HEAPU8.set(chunk, p);
            // write NAL units with AnnexB format
            // <Uint8Array [startcode] | [NAL] | [startcode] | [NAL] ...>
            MP4.mux_nal(mux, p, chunk.byteLength);
            MP4.free_buffer(p);
        }
    
        // Note: this may trigger more writes
        MP4.finalize_muxer(mux);
    
    
    
    })();
    
    opened by loretoparisi 3
  • get chunk data using copyTo method instead of EncodedVideoChunk.data

    get chunk data using copyTo method instead of EncodedVideoChunk.data

    data property in EncodedVideoChunk is internal according to the WebCodecs Spec and is not exposed by newer versions of Chrome.

    This may solve issue #3.

    I didn't test about AVCC since my Chrome seems not generating standard AVCC data. But it works excellently for AnnexB format in Chrome Beta (94.0.4606.41).

    opened by zhangbenber 2
  • Just wanted to say that this project is awesome

    Just wanted to say that this project is awesome

    Lots of apps can benefit from this type of WASM solution as opposed to bundling a full build of ffmpeg client-side or (even more work) adding to an async task queue for backend rendering.

    Great job, Matt -- feel free to close this issue 😄

    opened by transitive-bullshit 1
  • add options for addFrame, let keyFrame set by call.

    add options for addFrame, let keyFrame set by call.

    In some cases, we may want to force keyframe set of some specific frames. The commit add options for addFrame function, and make it possible.

    Thanks for review.

    opened by milkliker 0
  • Demuxer status

    Demuxer status

    Hi guys. This repo is very important initiative given the state of the web media capabilities )) I have got a question. What's the status of demuxing functionality? What's missing? Maybe I can help,because I an in search for a fast and minimal front end API for demuxing.

    Thanks.

    opened by sasmaster 0
  • Combining audio and video?

    Combining audio and video?

    First off, amazing package. Thank you.

    Was just wondering if you had any tips on combining audio with the encoded video? I've had success doing it with FFmpeg-wasm but it can't run on all devices.

    opened by Gnorme 0
  • Quality loss on macOS

    Quality loss on macOS

    Thanks for creating this project, it works surprisingly fast compared to other implementations.

    I'm using the webcodecs example where I draw on a canvas from a video element and then pass the bitmap image to the encoder.

    But there seems to be some quality loss over time on macOS. On Windows 10 it works just fine. There is a big difference in file size / bitrate too, see the examples bellow. Changing the bitrate or bitrateMode didn't change anything.

    Any idea what could cause this ?

    (I've also tried your other project, mp4-h264, which works but it's slower and couldn't make it work with webcodecs.)

    macOS 12.3.1 (MacBook Air M1, 2020) / Chrome 102 (filesize: 1.63 MB, bitrate: 910 kb/s)

    https://user-images.githubusercontent.com/1517945/174275139-be69b27a-a391-42a2-925c-f380d6b38824.mp4

    Windows 10 / Chrome 102 (filesize: 6.42 MB, bitrate: 3577 kb/s)

    https://user-images.githubusercontent.com/1517945/174265883-1d772d82-b262-46c7-b25f-71f35676407d.mp4

    MediaInfo

    macvswin

    opened by cretueusebiu 0
  • Cannot be used by bundlers

    Cannot be used by bundlers

    Currently, it's hard using this library directly from node_modules without copying over mp4.wasm.

    I am using esbuild for context.

    One easy solution would be to build with the emscripten flag -s SINGLE_FILE=1, that way the wasm binary code is encoded in the .js file. This results in a little bit filesize increase, but negligible overall.

    Another solution would be like ffmpeg.wasm which host the .wasm file by themselves.

    opened by marcofugaro 4
Owner
Matt DesLauriers
Matt DesLauriers
Minimalistic MP4 mux/demux single header library

Mini MP4 Easy embeddable MP4 mux/demux library. Usage Muxing Muxing can be done using 3 modes. Default mode uses one big mdat chunk: This is most effi

Lion 271 Dec 20, 2022
The Dolby MP4 streaming muxer (dlb_mp4base) is a software implementation of a muxer of fragmented or unfragmented ISO base media file format (mp4)

The Dolby MP4 streaming muxer (dlb_mp4base) is a software implementation of a muxer of fragmented or unfragmented ISO base media file format (mp4). It supports muxing of Dolby Digital (AC-3), Dolby Digital Plus (E-AC-3), and Dolby AC-4 audio formats as well as Dolby Vision.

Dolby Laboratories 203 Dec 21, 2022
The Dolby MP4 streaming demuxer (dlb_mp4demux) is a software implementation of a demuxer of fragmented or unfragmented ISO base media file format (mp4).

The Dolby MP4 streaming demuxer (dlb_mp4demux) is a software implementation of a demuxer of fragmented or unfragmented ISO base media file format (mp4). It supports demuxing of Dolby Digital (AC-3), Dolby Digital Plus (E-AC-3), and Dolby AC-4 audio formats as well as Dolby Vision. It is designed for use on architectures with limited resources.

Dolby Laboratories 68 Dec 20, 2022
A fast and small port of Zstandard to WASM.

Zstandard WASM A fast and small port of Zstandard to WASM. (Decompress-only for now). Features Fast: Zstandard has been compiled with the -03 flag, so

Fabio Spampinato 13 Nov 9, 2022
An Open Source PSVita/TV MP4 player with 1080p playback and subtitle support

Vita-Media-Player An Open Source PSVita/TV MP4 player with 1080p playback and subtitle support 1080i output supported on the PSTV natively and on the

Jaylon Gowie 50 Nov 25, 2022
simple mp4 player based on rockchip rv1109 platform

mp4player RV1109平台上实现一个简单的 mp4 播放器,主要是本人使用的开发板QT无法播放mp4,应该是没有编译qst所致,因而想利用rockchip平台自有的 功能实现一个简单的播放器。 base目录包含一些基础框架实现,包含信号,线程,时间等,线程和消息泵的实现非常非常简单,因而不

null 9 Jul 17, 2022
A project for the latest Steam PC (US) release of Resident Evil 4 that patches the game to use high-quality MP4 video.

A project for the latest Steam PC (US) release of Resident Evil 4 that patches the game to use high-quality MP4 video.

Megan Grass 6 Oct 10, 2022
This is a proof-of-concept of a modern C web-framework that compiles to WASM and is used for building user interfaces.

DanCing Web ?? ?? (DCW) Getting Started Dancing Web is now distributed with the Tarantella Package Manager — a tool I've made to simplify setup of pro

Danilo Chiarlone 3 Sep 11, 2021
A Nginx module which tries to implement proxy wasm ABI in Nginx.

Status This library is under construction. Description A Nginx module which tries to implement proxy wasm ABI in Nginx. Install dependencies Download

API7 104 Dec 29, 2022
An experiment of running a lot of wasm apps on esp32 M5 atom matrix board

Wasms This is a POC project to run many wasm apps on a single esp32 board (in my case, M5 Atom Matrix, with a nice neopixel 5x5 display matrix). All I

Dmitry Kabak 16 Sep 21, 2022
wwasm (Wgmlgz wasm) - is a c++ & reactjs liblary for connecting c++ backend and reactjs frontend.

WWASM (Wgmlgz wasm) - is a c++ & reactjs liblary for connecting c++ backend and reactjs frontend.

null 1 Nov 23, 2021
RTSP Wasm Player

RTSP Wasm Player Overview # RTSP WebSocket Proxy RTSP/Webcam/File > FFmpeg open > Packets > WebSocket # WS Wasm Player WebSocket > Packets > Wasm FFm

kuokuo 83 Dec 23, 2022
JS/WASM build of libjxl (JPEG-XL)

libjxl-js JS/WASM build of libjxl (JPEG-XL) Try It Out! Try it in your browser here Building This project uses git submodules to pull in libjxl. If de

Chris Hafey 12 Nov 28, 2022
Twitter thread explorer made in C++/Wasm/WebGL

Visa Viz Twitter thread explorer made in C++/Wasm/WebGL Once upon a time, I stumbled upon @visakanv's network of threads, but it was too cumbersome to

Ilia Demianenko 11 Oct 24, 2022
A single header C++ wasm frontend library leveraging Emscripten

Livid Livid is a single header C++ wasm frontend library leveraging Emscripten. Usage The code looks something like this: #include "livid/livid.hpp" #

Mohammed Alyousef 19 Nov 26, 2022
Toy path tracer for my own learning purposes (CPU/GPU, C++/C#, Win/Mac/Wasm, DX11/Metal, also Unity)

Toy Path Tracer Toy path tracer for my own learning purposes, using various approaches/techs. Somewhat based on Peter Shirley's Ray Tracing in One Wee

Aras Pranckevičius 931 Dec 29, 2022
Fast Binary Encoding is ultra fast and universal serialization solution for C++, C#, Go, Java, JavaScript, Kotlin, Python, Ruby, Swift

Fast Binary Encoding (FBE) Fast Binary Encoding allows to describe any domain models, business objects, complex data structures, client/server request

Ivan Shynkarenka 654 Jan 2, 2023
Peregrine - A blazing fast language for the blazing fast world(WIP)

A Blazing-Fast Language for the Blazing-Fast world. The Peregrine Programming Language Peregrine is a Compiled, Systems Programming Language, currentl

Peregrine 1.5k Jan 2, 2023
C++ implementation of a fast hash map and hash set using hopscotch hashing

A C++ implementation of a fast hash map and hash set using hopscotch hashing The hopscotch-map library is a C++ implementation of a fast hash map and

Thibaut Goetghebuer-Planchon 578 Dec 23, 2022
🏅State-of-the-art learned data structure that enables fast lookup, predecessor, range searches and updates in arrays of billions of items using orders of magnitude less space than traditional indexes

The Piecewise Geometric Model index (PGM-index) is a data structure that enables fast lookup, predecessor, range searches and updates in arrays of bil

Giorgio Vinciguerra 651 Dec 29, 2022