ShaderConductor is a tool designed for cross-compiling HLSL to other shading languages

Overview

ShaderConductor

Build Status License

ShaderConductor is a tool designed for cross-compiling HLSL to other shading languages.

Features

  • Converts HLSL to readable, usable and efficient GLSL
  • Converts HLSL to readable, usable and efficient ESSL
  • Converts HLSL to readable, usable and efficient Metal Shading Language (MSL)
  • Converts HLSL to readable, usable and efficient old shader model HLSL
  • Supports all stages of shaders, vertex, pixel, hull, domain, geometry, and compute.

Note that this project is still in an early stage, and it is under active development.

Architecture

ShaderConductor is not a real compiler. Instead, it glues existing open source components to do the cross-compiling.

  1. DirectX Shader Compiler to compile HLSL to DXIL or SPIR-V,
  2. SPIRV-Cross to convert SPIR-V to target shading languages.

Architecture

Prerequisites

  • Git. Put git into the PATH is recommended.
  • Visual Studio 2017. Select the following workloads: Universal Windows Platform Development and Desktop Development with C++.
  • CMake. Version 3.9 or up. It's highly recommended to choose "Add CMake to the system PATH for all users" during installation.
  • Python. Version 2.7 or up. You need not change your PATH variable during installation.

Building

ShaderConductor has been tested on Windows, Linux, and macOS.

The script way:

  BuildAll.py <BuildSystem> <Compiler> <Architecture> <Configuration>

where,

  • <BuildSystem> can be ninja or vs2017. Default is vs2017.
  • <Compiler> can be vc141 on Windows, gcc or clang on Linux, clang on macOS.
  • <Architecture> must be x64 (for now).
  • <Configuration> can be Debug, Release, RelWithDebInfo, or MinSizeRel. Default is Release.

This script automatically grabs external dependencies to External folder, generates project file in Build/<BuildSystem>-<Compiler>-<Platform>-<Architecture>[-<Configuration>], and builds it.

The manual way:

  mkdir Build
  cd Build
  cmake -G "Visual Studio 15" -T host=x64 -A x64 ../
  cmake --build .

After building, the output file ShaderConductor.dll can be located in <YourCMakeTargetFolder>/Bin/<Configuration>/. It depends on dxcompiler.dll in the same folder.

Artifacts

You can download the prebuilt binaries generated by CI system. Currently, artifacts for Windows, Linux, macOS are published every commit.

License

ShaderConductor is distributed under the terms of MIT License. See LICENSE for details.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Comments
  • Introduce option to inherit combined sampler bindings

    Introduce option to inherit combined sampler bindings

    This pull request introduces a new option that enables inheriting of combined sampler bindings.

    For an example, consider the original HLSL fragment:

    Texture2D tex0 : register(t0);
    Texture2D tex1 : register(t1);
    Texture2D tex2 : register(t2);
    Texture2D tex3 : register(t3);
    SamplerState sampler0 : register(s0);
    SamplerState sampler1 : register(s1);
    SamplerState sampler2 : register(s2);
    SamplerState sampler3 : register(s3);
    

    And the output with the option disabled (default/old behaviour):

    uniform sampler2D SPIRV_Cross_Combinedtex0sampler0;
    uniform sampler2D SPIRV_Cross_Combinedtex2sampler2;
    uniform sampler2D SPIRV_Cross_Combinedtex3sampler3;
    uniform sampler2D SPIRV_Cross_Combinedtex1sampler1;
    

    And compare with the output with the same option enabled:

    layout(binding = 0) uniform sampler2D SPIRV_Cross_Combinedtex0sampler0;
    layout(binding = 2) uniform sampler2D SPIRV_Cross_Combinedtex2sampler2;
    layout(binding = 3) uniform sampler2D SPIRV_Cross_Combinedtex3sampler3;
    layout(binding = 1) uniform sampler2D SPIRV_Cross_Combinedtex1sampler1;
    

    Note that in this particular (real world) example, the bindings changed. This is because SPIRV-Cross generates the combined samplers in the order of post-optimization usage, and not order of original binding or declaration.

    Without this option, it's impossible to write a generic system that will bind the GLSL textures to the same bindings as the original HLSL.

    This patch merely invokes the util method already provided by SPIRV-Cross.

    opened by amzeratul 6
  • Disassemble option

    Disassemble option

    Hi Minmin Gong,

    I added a new Disassemble option in TargetDesc to get disassemble Spirv for debug purposes. I hope that you find it useful to merge. Next step should be disassemble DXIL too.

    Note. I linked spirv-tools for this feature.

    Regards, Jorge

    enhancement 
    opened by Jorgemagic 6
  • Better legacy GLSL compatibility

    Better legacy GLSL compatibility

    Is your feature request related to a problem? Please describe. My engine supports GLES2 / WebGL1 in addition to newer OpenGL versions and Vulkan/DX12/Metal. By default, the shaders generated don't play well with the legacy GLSL versions. Since it's still a major target, I suggest some improvements & conventions to make it work out of the box.

    Describe the solution you'd like I suggest two improvements using SPRIVCross's shader reflection API:

    • Make the vertex output names match the fragment input names (renaming out_var_X and in_var_X to varying_X)
    • Append descriptor binding info into uniform names so it can be easily extracted when uniforms are enumerated after compilation/linking

    Describe alternatives you've considered The minor changes I propose should cover most of the use cases, but there might be a way to do this more generically with callbacks etc, although I feel it would go against the super streamlined design of ShaderConductor and would leak implementation details into user code.

    Additional context This is the code I'm using currently in ConvertBinary():

    case ShadingLanguage::Glsl:
    case ShadingLanguage::Essl:
        compiler = std::make_unique<spirv_cross::CompilerGLSL>(spirvIr, spirvSize);
        combinedImageSamplers = true;
        buildDummySampler = true;
    
        // Legacy GLSL fixups
        if (intVersion <= 300)
        {
            auto&& vars = compiler->get_active_interface_variables();
            for (auto& var : vars)
            {
                auto varClass = compiler->get_storage_class(var);
                
                // Make VS out and PS in variable names match
                if (source.stage == ShaderStage::VertexShader && varClass == spv::StorageClass::StorageClassOutput)
                {
                    auto name = compiler->get_name(var);
                    if (name.find("out_var") == 0)
                    {
                        name.replace(0, 7, "varying");
                        compiler->set_name(var, name);
                    }
                }
                else if (source.stage == ShaderStage::PixelShader && varClass == spv::StorageClass::StorageClassInput)
                {
                    auto name = compiler->get_name(var);
                    if (name.find("in_var") == 0)
                    {
                        name.replace(0, 6, "varying");
                        compiler->set_name(var, name);
                    }
                }
                
                // Encode binding info into variable name for uniform buffers, textures, samplers
                if (varClass == spv::StorageClass::StorageClassUniform || varClass == spv::StorageClass::StorageClassUniformConstant)
                {
                    auto space = compiler->get_decoration(var, spv::Decoration::DecorationDescriptorSet);
    				auto reg = compiler->get_decoration(var, spv::Decoration::DecorationBinding);
                
                    char buf[128];
                    sprintf(buf, "_reg_%d_space_%d", reg, space);
                
    				auto type = compiler->get_type_from_variable(var);
     				auto typeName = compiler->get_name(type.self);
     				typeName.append(buf);
     				compiler->set_name(type.self, typeName);
                
                    auto name = compiler->get_name(var);
                    name.append(buf);
                    compiler->set_name(var, name);
                }
            }
        }
    
        break;
    
    bug 
    opened by tklajnscek 4
  • libtinfo.so.5 dependency

    libtinfo.so.5 dependency

    Describe the bug

    The Linux CI builds produce a libdxcompiler.so build that dynamically links to libtinfo.so.5. That's not something that is always installed on a user machine. I have libtinfo.so.6 installed as an ncurses dependency on Arch Linux). I managed to run the CI binaries after installing ncurses5-compat-libs from the AUR. That provides libtinfo.so.5, though when running I get the following message /usr/lib/libtinfo.so.5: no version information available.

    Expected behavior

    There is no libtinfo dependeny.

    Additional context

    libtinfo is an optional dependency of LLVM, though I don't see where it is enabled exactly. It looks like it's only compiled as a dependency if the LLVM_ENABLE_TERMINFO flag is set. A cmake file in DirectXShaderCompiler checks that flag and finds a lib providing terminfo (e.g. libtinfo) here. The flag is enabled by default here. But it also looks like terminfo is disabled here. Building locally on my machine I don't get this dependency, but I'm unsure why.

    bug 
    opened by Jjagg 3
  • Add shift resource bindings in Vulkan

    Add shift resource bindings in Vulkan

    Hi @gongminmin ,

    I added 4 new parameters in Options struct:

    int shiftAllTexturesBindings: First binding number for Vulkan textures. int shiftAllSamplersBindings: First binding number for Vulkan samplers. int shiftAllCBuffersBindings: First binding number for Vulkan constant buffers. int shiftAllUABuffersBindings: First binding number for Vulkan unordered access buffers.

    It is equivalent to use -fvk-{b|s|t|u}-shift in DXC command-line. That is a very important option when you compile HLSL to SPIRV HLSL register and Vulkan binding.

    Finally, I inverted packMatricesInRowMajor option to fix the issue.

    Resolve #37 Resolve #41

    opened by Jorgemagic 2
  • Replace custom subrepository sync logic with git submodules

    Replace custom subrepository sync logic with git submodules

    This is related to #28 but not required for that issue. Instead of invoking git from inside CMake project generation this change uses git submodules to accomplish the same thing.

    opened by Qartar 2
  • CMake project generation requires git.exe to be in PATH

    CMake project generation requires git.exe to be in PATH

    Describe the bug CMake project generation requires git.exe to be in PATH in order to successfully run the UpdateExternalLib function defined in External/CMakeLists.txt. This requirement is not explicitly stated in README.md even though the instructions for CMake and Python both mention the PATH variable.

    To Reproduce

    1. Clone the repository
    2. In a context where git is not in PATH run Build.py or cmake -G ... as instructed in README.md

    Expected behavior Expected CMake project generation to complete successfully. Instead CMake silently fails when running the git commands in UpdateExternalLib and generates errors later when attempting to call add_subdirectory on the cloned external repositories and subsequent commands on targets in those repositories.

    Additional context I hit this issue because I have git installed into the PATH of Git Bash for Windows only and tried to generate the project files in the developer command prompt because of an issue with CMake and Visual Studio 2019.

    Not sure why git submodules aren't being used for this step.

    bug 
    opened by Qartar 2
  • Linux build failure due to -WError when building SPIRV-Tools

    Linux build failure due to -WError when building SPIRV-Tools

    Describe the bug SPIRV-Tools is compiled with the -WError flag. Building a clean clone of the repo with either clang or gcc fails because of the warning reported in KhronosGroup/SPIRV-Tools#1541. Not sure why CI builds don't run into this.

    To Reproduce Run python BuildAll.py ninja gcc or python BuildAll.py ninja clang. Clang version: 7.0.0 (tags/RELEASE_700/final) Gcc version: 8.2.1 20181127 (GCC)

    Expected behavior Cloning a build and running the python build script Just Works.

    Additional context I don't think ShaderConductor needs to set -WError. This can be disabled by setting SPIRV_WERROR=OFF.

    bug 
    opened by Jjagg 2
  • Investigate why dxcompiler file is so large on Linux and macOS

    Investigate why dxcompiler file is so large on Linux and macOS

    dxcompiler.dll is 17.7MB (7.7MB compressed), but libdxcompiler.so is 34.5MB (13.1MB compressed), and libdxcompiler.dylib is 27.1MB (10.0MB compressed). Need to have some understanding about why they are so different, and how to improve this.

    enhancement 
    opened by gongminmin 2
  • CSharpWrapper

    CSharpWrapper

    I am using ShaderConductor library from C# so I created a wrapper project and C# pinvoke test project. I hope that you find it interesting to accept in your repo.

    opened by Jorgemagic 2
  • Static Library support

    Static Library support

    Is your feature request related to a problem? Please describe. If I want to create a portable tool that has shader conversion support, or if I want to add the ability to generate shader code to a library, I need to worry about the DLLs.

    Describe the solution you'd like Use of BUILD_SHARED_LIBS in cmake or something. Whatever is easiest to implment.

    Describe alternatives you've considered N/A

    Additional context N/A

    opened by Tetrago 1
  • This repo is missing important files

    This repo is missing important files

    There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

    Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

    Merge this pull request

    opened by microsoft-github-policy-service[bot] 1
  • Adding Microsoft SECURITY.MD

    Adding Microsoft SECURITY.MD

    Please accept this contribution adding the standard Microsoft SECURITY.MD :lock: file to help the community understand the security policy and how to safely report security issues. GitHub uses the presence of this file to light-up security reminders and a link to the file. This pull request commits the latest official SECURITY.MD file from https://github.com/microsoft/repo-templates/blob/main/shared/SECURITY.md.

    Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

    opened by microsoft-github-policy-service[bot] 0
  • Pre-compiled binaries not available currently

    Pre-compiled binaries not available currently

    opened by SomeAB 0
  • [Reflection] Implement SPIR-V shader reflection

    [Reflection] Implement SPIR-V shader reflection

    Note that this PR is on top of 69, 70, 71, and 72. Please review them first. This change needs to be rebased after those PRs go in. After this PR, we have both DXIL and SPIR-V reflections.

    opened by gongminmin 0
Releases(v0.3.0)
Owner
Microsoft
Open source projects and samples from Microsoft
Microsoft
SPIRV-Cross is a tool designed for parsing and converting SPIR-V to other shader languages.

SPIRV-Cross SPIRV-Cross is a tool designed for parsing and converting SPIR-V to other shader languages. Features Convert SPIR-V to readable, usable an

The Khronos Group 1.6k Nov 22, 2022
A Visual Studio extension that provides enhanced support for editing High Level Shading Language (HLSL) files

HLSL Tools for Visual Studio This extension is for Visual Studio 2017 / 2019. Go here for the Visual Studio Code extension. HLSL Tools is a Visual Stu

Tim Jones 427 Nov 22, 2022
HLSL Parser and Translator for HLSL, GLSL, and MSL.

HLSLParser This is a fork of Unknownworld's hlslparser adapted to our needs in The Witness. We currently use it to translate pseudo-HLSL shaders (usin

null 3 Jul 2, 2022
HLSL Parser and Translator for HLSL, GLSL, and MSL.

HLSLParser This is a fork of Unknownworld's hlslparser adapted to our needs in The Witness. We currently use it to translate pseudo-HLSL shaders (usin

null 316 Nov 15, 2022
Guide to Cross Compiling on a Raspberry Pi

Guide to Cross Compilation for a Raspberry Pi > Start Setup XCS and RPi Setup RPi Network and SSH Setup RPi Peripherals Setup Cross-compile environmen

Hessel van der Molen 54 Oct 4, 2022
Notepad++ is a free source code editor and Notepad replacement that supports several programming languages and natural languages

Npp / Notepad++ is my customized text editor highly enhanced for coding such as insta-run, much more file extensions made self-recognizable, logically colored syntax highlighting for nearly every programming language and designed for very easy customizability -- from the toolbar, context menu, syntax coloring, plug-ins for optional increased capabilities and much more

SkyN9ne 1 Jan 23, 2022
PoC tool to coerce Windows hosts to authenticate to other machines via MS-EFSRPC EfsRpcOpenFileRaw or other functions.

PetitPotam PoC tool to coerce Windows hosts to authenticate to other machines via MS-EFSRPC EfsRpcOpenFileRaw or other functions :) The tools use the

Topotam 1.4k Nov 23, 2022
Mix C with other programming languages

extern "C" This project demonstrates how to mix C with other programming languages. It aims to create a Rosetta Stone of the procedure shown in the fo

Ralph 12 Mar 25, 2022
Jaws is an invisible programming language! Inject invisible code into other languages and files! Created for security research -- see blog post

Jaws is an invisible interpreted programming language that was created for antivirus research. Since Jaws code is composed entirely of whitespace char

C.J. May 207 Oct 17, 2022
Write snippets of C code in your txt files for notes and skip the hassle of compiling and running

Write snippets of C code in your txt files for notes and skip the hassle of compiling and running. Greatly helps organization and note-taking to make sure you do not miss anything.

Seamus Walden 4 Jun 13, 2022
glslcc: Cross-compiler for GLSL shader language (GLSL->HLSL,METAL,GLES,GLSLv3)

glslcc: Cross-compiler for GLSL shader language (GLSL->HLSL,METAL,GLES,GLSLv3) @septag glslcc is a command line tool that converts GLSL code to HLSL,

Sepehr Taghdisian 432 Nov 24, 2022
Shader cross compiler to translate HLSL (Shader Model 4 and 5) to GLSL

XShaderCompiler ("Cross Shader Compiler") Features Cross compiles HLSL shader code (Shader Model 4 and 5) into GLSL Simple to integrate into other pro

Lukas Hermanns 346 Oct 1, 2022
Light probe generation and BRDF authoring for physically based shading.

IBLBaker About IBLBaker is provided under the MIT License(MIT) Copyright(c) 2015 Matt Davidson. Please see the LICENSE file for full details. Feel fre

MattD 657 Nov 16, 2022
An implementation of physically based shading & image based lighting in D3D11, D3D12, Vulkan, and OpenGL 4.

Physically Based Rendering (c) 2017 - 2018 Michał Siejak (@Nadrin) An implementation of physically based shading model & image based lighting in vario

Michał Siejak 1.1k Nov 15, 2022
Unix pager (with very rich functionality) designed for work with tables. Designed for PostgreSQL, but MySQL is supported too. Works well with pgcli too. Can be used as CSV or TSV viewer too. It supports searching, selecting rows, columns, or block and export selected area to clipboard.

Unix pager (with very rich functionality) designed for work with tables. Designed for PostgreSQL, but MySQL is supported too. Works well with pgcli too. Can be used as CSV or TSV viewer too. It supports searching, selecting rows, columns, or block and export selected area to clipboard.

Pavel Stehule 1.9k Nov 22, 2022
『HLSL シェーダーの魔導書』(ISBN978-4-7981-6428-1)のサンプルファイル

# サンプルデータについて 本データは、『HLSL シェーダーの魔導書』(清原 隆行 著、翔泳社 刊)の付属データです。 なお、本データは以下のサイトから入手可能です。 - Github:https://github.com/shoeisha-books/hlsl-grimoire-sampl

SEBook 翔泳社の本 98 Nov 18, 2022
HLSL to GLSL language translator based on ATI's HLSL2GLSL. Used in Unity.

HLSL to GLSL shader language translator ⚠️ As of mid-2016, the project is unlikely to have any significant developments. At Unity we are moving to a d

Aras Pranckevičius 521 Oct 5, 2022
Minify and obfuscate GLSL or HLSL code

Shader Minifier Shader Minifier is a tool that minifies and obfuscates shader code (GLSL and HLSL). Its original use-case is for the demoscene, for op

Laurent Le Brun 245 Nov 27, 2022
Khronos-reference front end for GLSL/ESSL, partial front end for HLSL, and a SPIR-V generator.

News Visual Studio 2013 is no longer supported As scheduled, Microsoft Visual Studio 2013 is no longer officially supported. Please upgrade to at leas

The Khronos Group 2.3k Nov 24, 2022