HLSL Parser and Translator for HLSL, GLSL, and MSL.

Overview

HLSLParser

This is a fork of Unknownworld's hlslparser adapted to our needs in The Witness. We currently use it to translate pseudo-HLSL shaders (using the legacy D3D9 syntax) to HLSL10 and Metal Shading Language (MSL). There's also a GLSL translator available that we do not use yet, but that is being maintained by community contributions.

The HLSL parser has been extended with many HLSL10 features, but retaining the original HLSL C-based syntax.

For example, the following functions in our HLSL dialect:

float tex2Dcmp(sampler2DShadow s, float3 texcoord_comparevalue);
float4 tex2DMSfetch(sampler2DMS s, int2 texcoord, int sample);
int2 tex2Dsize(sampler2D s);

Are equivalent to these methods in HLSL10:

float Texture2D::SampleCmp(SamplerComparisonState s, float2 texcoord, float comparevalue);
float4 Texture2DMS<float4>::Load(int2 texcoord, int sample);
void Texture2D<float4>::GetDimensions(out uint w, out uint h);

Here are the original release notes:

HLSL Parser and GLSL code generator

This is the code we used in Natural Selection 2 to convert HLSL shader code to GLSL for use with OpenGL. The code is pulled from a larger codebase and has some dependencies which have been replaced with stubs. These dependencies are all very basic (array classes, memory allocators, etc.) so replacing them with our own equivalent should be simple if you want to use this code.

The parser is designed to work with HLSL code written in the legacy Direct3D 9 style (e.g. D3DCOMPILE_ENABLE_BACKWARDS_COMPATIBILITY should be used with D3D11). The parser works with cbuffers for uniforms, so in addition to generating GLSL, there is a class provided for generating D3D9-compatible HLSL which doesn't support cbuffers. The GLSL code requires version 3.1 for support of uniform blocks. The parser is designed to catch all errors and generate "clean" GLSL which can then be compiled without any errors.

The HLSL parsing is done though a basic recursive descent parser coded by hand rather than using a parser generator. We believe makes the code easier to understand and work with.

To get consistent results from Direct3D and OpenGL, our engine renders in OpenGL "upside down". This is automatically added into the generated GLSL vertex shaders.

Although this code was written specifically for our use, we hope that it may be useful as an educational tool or a base for someone who wants to do something similar.

Issues
  • MSL: Change tex2DMSfetch signature to use int2 inputs

    MSL: Change tex2DMSfetch signature to use int2 inputs

    This fixes a compilation issue with passing int2 to tex2Dmsfetch - Metal compiler can't pick the right overload.

    It's possible to fix this by also adding an int2 overload but I felt that just using int2 makes more sense - this matches GLSL and DX11 semantics for fetching (e.g. https://msdn.microsoft.com/en-us/library/windows/desktop/bb509694(v=vs.85).aspx).

    opened by zeux 4
  • Many GLSL and MSL fixes and improvements

    Many GLSL and MSL fixes and improvements

    Let me preface this by saying that since Thekla/hlslparser is sort of Witness specific, feel free to reject my pull request if you think you guys don't need these changes :) I also apologize for the size of the request - this is an accumulation of many commits in my repository but since I had to do pretty crazy merges from Thekla with whitespace changes etc. I don't know if I can split it into small chunks easily. Finally, while this is mostly strict improvements, it does change the output shaders a bit wrt the contract between the API and shaders - I summarized what I think are breaking changes at the end.

    Here's the summary of all changes this contains:

    General code improvements:

    • Various warning fixes
    • Split WriteLineTagged from WriteLine for file/line overloads - existing overload makes it impossible to use "%s %d" format since the file/line overload is mistakenly selected
    • More cleanly add printf attributes
    • Add Travis CI config and a Makefile
    • Expand command-line tool to support more output languages
    • Fix VS2012 compatibility - do not rely on field initializers inside struct definitions

    GLSL improvements/fixes:

    • Fix #line directives to avoid the file name which does not conform to the standard
    • Add support for various GLSL versions (GLSL 2 & 3, GLSL ES 2 & 3). This required tweaks throughout the codegen since the resulting syntax is frequently somewhat different.
    • Add texture array support
    • Use more precise filtering for the output functions - instead of checking if string pool contains a name check if there's a function that is used
    • Implement implicit casting (HLSL-like) in ternary ifs and some other operators
    • Implement support for MRT color output and generally improve built-in semantic support (add D3D9 semantics like POSITION, add PSIZE, VPOS, etc.)
    • Improve attribute/varying generation - code now works with output arguments (out float4 pos: POSITION), better struct support, etc.
    • Move gl_Position flipping behind a separate codegen flag (FlipPosition)
    • Add a flag to switch between column-major and row-major matrices - currently only affects matrix multiplication...
    • Add a flag to emulate constant buffers - this generates a flat uniform array vec4 ...[N] and generates optimal statically or dynamically computed indices into that. This helps with compiling for GL 2/ GLES 2.
    • Add a flag to lower matrix multiplication to dot products or mul/mad - this improves codegen after glsl-optimizer because it's able to better simplify certain expressions
    • Add tex2Dgrad support
    • Add ddx/ddy support
    • Add function forward declaration support

    MSL improvements/fixes:

    • General translation for constant buffers instead of hardcoding per_item/etc.
    • Add support for explicit sampler and constant buffer register specification (: register(s1), etc.)
    • Some fixes for output filtering based on whether we think we need certain types/functions - rely on NeedsFunction and fix some corner cases
    • Add function forward declaration support
    • Add float4x4-to-float3x3 cast support
    • Add all pairs of square-matrix-by-vector multiplication

    Likely regressions on existing shaders:

    • Took out float3x4/float4x3 support from MSL and GLSL. This is probably a mistake and I can restore it.
    • Restructured hard-coded MSL vertex attribute mapping. This is probably something that needs support for external configuration
    • Default GLSL flags disable gl_Position flipping. I believe this is the right call and caller should specify flipping if it is needed
    • MSL shaders may get bigger as far as output source code is concerned unless the tree is pruned using PruneTree. I believe this is the right call.
    • Removed EmulateAlphaTest. This is probably a mistake - I forgot the reason why I did this :) I can restore it.
    opened by zeux 4
  • MSL: Fix 'static const' variables

    MSL: Fix 'static const' variables

    Static const variables are not supported in function scope (because of Metal restrictions), and also aren't supported in class scope (C++ linking rules).

    For them to work, we need to lift them all the way up to global scope because of this.

    This lifting is somewhat naive - it doesn't resolve naming conflicts and does not lift struct declarations - but it's better than nothing...

    opened by zeux 1
  • GLSL: Implement support for static const

    GLSL: Implement support for static const

    In GLSL 'const' means 'static const'; this change fixes translation of static const variables by removing the uniform keyword and adding const instead. This means that static const variables are automatically set when the shader is bound and can be optimized by the driver, which matches HLSL behavior.

    opened by zeux 1
  • MSL: Add support for texture arrays

    MSL: Add support for texture arrays

    We add 0.5 to layer index to match round-to-nearest semantics expected by HLSL code.

    Without doing that, on NVidia GPUs rasterizer can have a slight rounding error if you pass the texture array layer via interpolator, and the implicit cast inside sample call would truncate something like 3.999999 to 3.

    opened by zeux 1
  • Fix CB emulation when copying the struct out of CB

    Fix CB emulation when copying the struct out of CB

    When an entire struct was copied out of the uniform buffer, like so:

    Foo foo = global_ubo.foos[3];

    We didn't propagate the struct type correctly; this change fixes it.

    opened by zeux 1
  • Fixed from PVS-Studio Static Analyzer

    Fixed from PVS-Studio Static Analyzer

    I'm a member of the Pinguem.ru competition on finding errors in open source projects. A bug, found using PVS-Studio. Warnings:

    V796 It is possible that 'break' statement is missing in switch statement. hlsltokenizer.cpp 625

    opened by PufonBarn 1
  • fixed from PVS-Studio

    fixed from PVS-Studio

    I'm a member of the Pinguem.ru competition on finding errors in open source projects. A bug, found using PVS-Studio. Warnings:

    V796 It is possible that 'break' statement is missing in switch statement. hlsltokenizer.cpp 625

    opened by ip-gpu 1
  • fixed from PVS-Studio

    fixed from PVS-Studio

    I'm a member of the Pinguem.ru competition on finding errors in open source projects. A bug, found using PVS-Studio. Warnings:

    V668 There is no sense in testing the 'res' pointer against null, as the memory was allocated using the 'new' operator. The exception will be generated in the case of memory allocation error. engine.cpp 129

    opened by ip-gpu 1
  • Merge changes from The Witness for iOS.

    Merge changes from The Witness for iOS.

    • Added better support for out and inout arguments in the MSL generator. Instead of passing these by reference, output arguments are wrapped in a temporary structure that's passed as return value. To make this work, the corresponding function call arguments need to be flattened and transformed into a list of statements.
    • Added support for sample_id semantic.
    • Added support for half, with an option to treat half as float in MSL. Support for half required better handling of type casts, since in many cases casts were not implicit as in HLSL.
    • Added support for treating integers as 16 bit integers (int -> short, uint->usrhot).
    • The generated MSL output is a bit cleaner and some unnecessary parenthesis have been removed.
    • Fixed support for some const declarations.
    • There's some work in progress to add support for static if and some new hlsl features that didn't make it for the iOS release.
    opened by castano 0
  • MSL: Remove 2-arity tex2Dmsfetch

    MSL: Remove 2-arity tex2Dmsfetch

    When merging our fork with this one one of the shaders that uses tex2Dmsfetch didn't compile for MSL for us because one of the overloads was invalid. I think it's a regression from 4221154909ef837b08c882a714dfb64e379b7695.

    This change just removes it; this assumes that The Witness does not actually use this overload :) If you guys do I could rework this change somehow.

    opened by zeux 0
HLSL to GLSL language translator based on ATI's HLSL2GLSL. Used in Unity.

HLSL to GLSL shader language translator ⚠️ As of mid-2016, the project is unlikely to have any significant developments. At Unity we are moving to a d

Aras Pranckevičius 512 Jun 21, 2022
glslcc: Cross-compiler for GLSL shader language (GLSL->HLSL,METAL,GLES,GLSLv3)

glslcc: Cross-compiler for GLSL shader language (GLSL->HLSL,METAL,GLES,GLSLv3) @septag glslcc is a command line tool that converts GLSL code to HLSL,

Sepehr Taghdisian 405 Jun 23, 2022
⚔️ A tool for cross compiling shaders. Convert between GLSL, HLSL, Metal Shader Language, or older versions of GLSL.

A cross compiler for shader languages. Convert between SPIR-V, GLSL / GLSL ES, HLSL, Metal Shader Language, or older versions of a given language. Cross Shader wraps glslang and SPIRV-Cross, exposing a simpler interface to transpile shaders.

Alain Galvan 157 Jun 23, 2022
LLVM IR and optimizer for shaders, including front-end adapters for GLSL and SPIR-V and back-end adapter for GLSL

Licensing LunarGLASS is available via a three clause BSD-style open source license. Goals The primary goals of the LunarGLASS project are: Reduce the

LunarG, Inc. 151 Jun 18, 2022
GLSL optimizer based on Mesa's GLSL compiler. Used to be used in Unity for mobile shader optimization.

GLSL optimizer ⚠️ As of mid-2016, the project is unlikely to have any significant developments. At Unity we are moving to a different shader compilati

Aras Pranckevičius 1.5k Jun 23, 2022
Minify and obfuscate GLSL or HLSL code

Shader Minifier Shader Minifier is a tool that minifies and obfuscates shader code (GLSL and HLSL). Its original use-case is for the demoscene, for op

Laurent Le Brun 219 Jun 23, 2022
Shader cross compiler to translate HLSL (Shader Model 4 and 5) to GLSL

XShaderCompiler ("Cross Shader Compiler") Features Cross compiles HLSL shader code (Shader Model 4 and 5) into GLSL Simple to integrate into other pro

Lukas Hermanns 342 Jun 20, 2022
Khronos-reference front end for GLSL/ESSL, partial front end for HLSL, and a SPIR-V generator.

News Visual Studio 2013 is no longer supported As scheduled, Microsoft Visual Studio 2013 is no longer officially supported. Please upgrade to at leas

The Khronos Group 2.2k Jun 25, 2022
exp2swift translator which translates STEP schema files described in ISO 10303-11 STEP EXPRESS language to the Swift programing language.

STEPswiftcode/ exp2swift exp2swift translator which translates STEP schema files described in ISO 10303-11 STEP EXPRESS language to the Swift programi

Tsutomu Yoshida 1 Jan 3, 2022
Fast C/C++ CSS Parser (Cascading Style Sheets Parser)

MyCSS — a pure C CSS parser MyCSS is a fast CSS Parser implemented as a pure C99 library with the ability to build without dependencies. Mailing List:

Alexander 119 Jun 18, 2022
this is my simple voxel engine, appart from librairies like glad it is entierly written in C++ and GLSL

simple-voxel-raycaster this is my simple voxel engine, appart from librairies like glad it is entierly written in C++ and GLSL here is a gif: https://

null 1 Jun 4, 2022
glsl code blocks for org-mode

GLSL code blocks for Emacs Org-mode This org-mode extension adds the capability to run GLSL code blocks directly from inside Emacs and immediately dis

null 27 Jun 14, 2022
VSIX Project that provides GLSL language integration.

GLSL language integration (for VS2017, 2019 and 2022) Download this extension from Visual Studio Marketplace version VS2017 & 2019 or VS 2022 preview

Daniel Scherzer 193 Jun 22, 2022
Simple printf functionality for GLSL.

Simple printf functionality for GLSL. This library is a simple proof of concept of using printf directly from a shader. The main point of being able

null 200 Jun 19, 2022
An efficient texture-free GLSL procedural noise library

Wombat An efficient texture-free GLSL procedural noise library Source: https://github.com/BrianSharpe/Wombat Derived from: https://github.com/BrianSha

Brian Sharpe 192 Jun 21, 2022
Fast glsl deNoise spatial filter, with circular gaussian kernel, full configurable

glslSmartDeNoise Fast glsl spatial deNoise filter, with circular gaussian kernel and smart/flexible/adaptable -> full configurable: Standard Deviation

Michele Morrone 190 Jun 26, 2022
『HLSL シェーダーの魔導書』(ISBN978-4-7981-6428-1)のサンプルファイル

# サンプルデータについて 本データは、『HLSL シェーダーの魔導書』(清原 隆行 著、翔泳社 刊)の付属データです。 なお、本データは以下のサイトから入手可能です。 - Github:https://github.com/shoeisha-books/hlsl-grimoire-sampl

SEBook 翔泳社の本 81 Jun 26, 2022
A Visual Studio extension that provides enhanced support for editing High Level Shading Language (HLSL) files

HLSL Tools for Visual Studio This extension is for Visual Studio 2017 / 2019. Go here for the Visual Studio Code extension. HLSL Tools is a Visual Stu

Tim Jones 398 Jun 22, 2022
ShaderConductor is a tool designed for cross-compiling HLSL to other shading languages

ShaderConductor ShaderConductor is a tool designed for cross-compiling HLSL to other shading languages. Features Converts HLSL to readable, usable and

Microsoft 1.4k Jun 28, 2022