SMOL-V: like Vulkan/Khronos SPIR-V, but smaller.

Overview

SMOL-V: like Vulkan/Khronos SPIR-V, but smaller. Build Status

Overview

SMOL-V encodes Vulkan/Khronos SPIR-V format programs into a form that is smoller, and is more compressible. Normally no changes to the programs are done; they decode into exactly same program as what was encoded. Optionally, debug information can be removed too.

SPIR-V is a very verbose format, several times larger than same programs expressed in other shader formats (e.g. DX11 bytecode, GLSL, DX9 bytecode etc.). The SSA-form with ever increasing IDs is not very appreciated by regular data compressors either. SMOL-V does several things to improve this:

  • Many words, especially ones that most often have small values, are encoded using "varint" scheme (1-5 bytes per word, with just one byte for values in 0..127 range).
  • Some IDs used in the program are delta-encoded, relative to previously seen IDs (e.g. Result IDs). Often instructions reference things that were computed just before, so this results in small deltas. These values are also encoded using "varint" scheme.
  • Reordering instruction opcodes so that the most common ones are the smallest values, for smaller varint encoding.
  • Encoding several instructions in a more compact form, e.g. the "typical <=4 component swizzle" shape of a VectorShuffle instruction, or sequences of MemberDecorate instructions.

A somewhat similar utility is spirv-remap from glslang.

See this blog post for more information about how I did SMOL-V.

Usage

Add source/smolv.h and source/smolv.cpp to your C++ project build. It might require C++11 or somesuch. There are Github Actions set up for this project, that build on Windows (VS2017), macOS (Xcode 11.1) and Linux (Ubuntu 16 / gcc 5.4).

smolv::Encode and smolv::Decode is the basic functionality. See smolv.h.

Other functions are for development/statistics purposes, to figure out frequencies and distributions of the instructions.

There's a test + compression benchmarking suite in testing/testmain.cpp, using that needs adding other files under testing/external to the build too (3rd party code: glslang remapper, Zstd, LZ4, miniz).

Changelog

See Changelog.

Limitations / TODO

  • SPIR-V where the words got stored in big-endian layout is not supported yet.
  • The whole thing might not work on Big-Endian CPUs. It might, but I'm not 100% sure.
  • Not much prevention is done against malformed/corrupted inputs, TODO.
  • Out of memory cases are not handled. The code will either throw exception or crash, depending on your compilation flags.

License

Code itself: Public Domain or MIT, pick whichever works better for you.

There is 3rd party code under the testing framework (testing/external); it is not required for using SMOL-V. Most of that code (glslang, LZ4, Zstd, sokol_time.h) is BSD or zlib-licensed, and taken from github repositories of the respective projects. miniz is public domain.

There are SPIR-V binary shader dumps under tests/spirv-dumps for compression testing; these are not required for using SMOL-V. Not sure how to appropriately "license" them (but hey they are kinda useless by themselves out of context), so I'll go with this: "Binary shader dumps under 'tests' folder are only to be used for SMOL-V testing". Details on them:

  • tests/spirv-dumps/dota2 - some shaders from DOTA2, Copyright Valve Corporation, all rights reserved.
  • tests/spirv-dumps/shadertoy - several most popular shaders from Shadertoy, converted to Vulkan SPIR-V via glslang. Copyrights by their individual authors (filename matches last component of shadertoy URL).
  • tests/spirv-dumps/talos - some shaders from The Talos Principle, Copyright (c) 2002-2016 Croteam All rights reserved.
  • tests/spirv-dumps/unity - various Unity shaders, produced through a HLSL -> DX11 bytecode -> HLSLcc -> glslang toolchain.

Results

As of 2020 May 25 29, results on 372 shaders (under tests/spirv-dumps) are:

Compressed with <none>:
Raw        5188.9KB 100.0%
Remapper   5089.0KB  98.1%
SmolV      1934.2KB  37.3%

Compressed with zlib:
Raw        1301.3KB  25.1%
Remapper   1230.5KB  23.7%
SmolV       696.6KB  13.4%

Compressed with LZ4 HC:
Raw        1448.7KB  27.9%
Remapper   1303.9KB  25.1%
SmolV       711.3KB  13.7%

Compressed with Zstandard:
Raw         983.4KB  19.0%
Remapper    870.8KB  16.8%
SmolV       541.3KB  10.4%

Compressed with Zstandard 20:
Raw         649.6KB  12.5%
Remapper    599.1KB  11.5%
SmolV       419.7KB   8.1%

Decoding these 372 shaders from SMOL-V back into SPIR-V takes 10.2ms (VS2017, x64 Release, AMD ThreadRipper 1950X 3.4GHz, one thread).

  • "Raw" is just raw SPIR-V, with no extra processing.
  • "Remapper" is spirv-remap from glslang, with debug info stripping.
  • SmolV is what you're looking at, with debug info stripping too.
  • zlib, LZ4HC and Zstd are general compression algorithms at default settings (Zstd20 is Zstd compression with almost max setting of 20).
  • Compression is done on the whole blob of all the test programs (not individually for each program).
You might also like...
Khepri is a Cross-platform agent, the architecture and usage like Coblat Strike but free and open-source.
Khepri is a Cross-platform agent, the architecture and usage like Coblat Strike but free and open-source.

Khepri Free,Open-Source,Cross-platform agent and Post-exploiton tool written in Golang and C++ Description Khepri is a Cross-platform agent, the archi

This repo does not contain any skins that work by themselves, but rather addons to already existing skins like CakeOS and Polybar
This repo does not contain any skins that work by themselves, but rather addons to already existing skins like CakeOS and Polybar

Rainmeter-addons ⚠ This repo does not contain any skins that work by themselves, but rather addons to already existing skins like CakeOS and Polybar E

It's like Porth, but in C++. Yep, we're going full circle.

Corth It's like Porth, which is like Forth but written in Python, but written in C++. But I don't actually know for sure since I never programmed in P

Like APL, but with emojis

🙂 APL This idea has been holding my brain hostage, I hope that implementing it will restore my free will. 🗨️ The canonical game of life function, ca

Vimb - the vim like browser is a webkit based web browser that behaves like the vimperator plugin for the firefox and usage paradigms from the great editor vim.

Vimb - the vim like browser is a webkit based web browser that behaves like the vimperator plugin for the firefox and usage paradigms from the great editor vim. The goal of vimb is to build a completely keyboard-driven, efficient and pleasurable browsing-experience.

A ZX Spectrum-like library built for
A ZX Spectrum-like library built for "dos-like" by Mattias Gustavsson.

ZX-Like A ZX Spectrum-like library built for "dos-like" by Mattias Gustavsson. It allows for the creation of ZX Spectrum like screens for demos, games

layer to control the global priority of any vulkan application

vk-force-priority vk-force-priority allows you to control the global priority of any vulkan application. Building from Source Dependencies Before buil

Collection of Vulkan samples
Collection of Vulkan samples

This repository holds many samples, showing various aspect of Vulkan, debugging tips and usage of other Nvidia tools.

Vulkan and other GPU API bugs I found.
Vulkan and other GPU API bugs I found.

GPU-my-list-of-bugs what is it - list of bugs I found writing shaders, mostly shader bugs. Maybe this is my code bug or/and shader bugs, but this code

Comments
  • Added

    Added "versioning" i.e. backwards compat for old encoded shaders, as SMOL-V learns about new SPIR-V opcodes

    SMOL-V can encode SPIR-V instructions that it knows nothing about yet (from various SPIR-V extensions etc.). That is not super efficient, but it works.

    However, if at some point in the future SMOL-V is updated to "know" about these instructions properly, then we need a way to decode both the previously encoded shaders, and the now-encoded shaders.

    This adds the capability, by using one unused byte in the header version word. Currently only two SMOL-V versions: 0 (the one ever existed before) and 1 (with new instructions added 10 days ago on 2020 Feb).

    However, this means SMOL-V shaders encoded during last 10 days that happen to use the new instructions (mostly group operations), need to be re-encoded.

    opened by aras-p 0
  • Signed/Unsigned compare warning

    Signed/Unsigned compare warning

    https://github.com/aras-p/smol-v/blob/acd27e677cc45b4a815e79b72beec4a92a0589fc/source/smolv.cpp#L1396

    Hey, this is just a super minor stuff. There a 3 places [lines: 1396, 1578 and 1781 ] where VS17 with -W4 shoots some warning. I'm using pedantic -treat-warnings-as-errors flag, so that was not compiling. Would be convinient to replace int i to unsigned i .

    Thanks! Great lib btw ;)

    opened by sagaceilo 0
Owner
Aras Pranckevičius
Code plumber at Unity
Aras Pranckevičius
A faster,smaller,Address Sanitizer,200X Faster,95% Smaller.

FirASAN(Fir Address Sanitizer) A faster,smaller,Address Sanitizer 200X Faster,95% Smaller. FirASAN 结论: 内存消耗 CPU消耗 ASAN原版 +100-150% +200%以上 FirASAN +5%

null 11 Aug 16, 2022
Khronos-reference front end for GLSL/ESSL, partial front end for HLSL, and a SPIR-V generator.

News Visual Studio 2013 is no longer supported As scheduled, Microsoft Visual Studio 2013 is no longer officially supported. Please upgrade to at leas

The Khronos Group 2.3k Sep 18, 2022
The SPIR-V Tools project provides an API and commands for processing SPIR-V modules.

SPIR-V Tools Overview The SPIR-V Tools project provides an API and commands for processing SPIR-V modules. The project includes an assembler, binary m

The Khronos Group 781 Sep 19, 2022
MozJPEG improves JPEG compression efficiency achieving higher visual quality and smaller file sizes at the same time

Mozilla JPEG Encoder Project MozJPEG improves JPEG compression efficiency achieving higher visual quality and smaller file sizes at the same time. It

Mozilla 5k Sep 20, 2022
SPIRV-Cross is a tool designed for parsing and converting SPIR-V to other shader languages.

SPIRV-Cross SPIRV-Cross is a tool designed for parsing and converting SPIR-V to other shader languages. Features Convert SPIR-V to readable, usable an

The Khronos Group 1.5k Sep 14, 2022
LLVM IR and optimizer for shaders, including front-end adapters for GLSL and SPIR-V and back-end adapter for GLSL

Licensing LunarGLASS is available via a three clause BSD-style open source license. Goals The primary goals of the LunarGLASS project are: Reduce the

LunarG, Inc. 154 Sep 12, 2022
This repository contains machine-readable files for the SPIR-V Registry

SPIR-V Headers This repository contains machine-readable files for the SPIR-V Registry. This includes: Header files for various languages. JSON files

The Khronos Group 191 Sep 19, 2022
yariv.h is a single C/C++ header to encode and decode SPIR-V shaders into a more compressed form I've called YARI-V.

YARI-V yariv.h is a single C/C++ header to encode and decode SPIR-V shaders into a more compressed form I've called YARI-V. YARI-V is an alternative e

Neil Henning 32 May 3, 2022
The Pizza Compass will determine your location and direct you to the nearest pizza place. It’s like a regular compass, but better!

Pizza_Compass A Particle project named Pizza_Compass Welcome to your project! Every new Particle project is composed of 3 important elements that you'

Joe Grand 68 Aug 16, 2022