SPIRV-Cross is a tool designed for parsing and converting SPIR-V to other shader languages.

The Khronos Group

Last update: Jan 2, 2023

Related tags

Miscellaneous SPIRV-Cross

Overview

SPIRV-Cross

SPIRV-Cross is a tool designed for parsing and converting SPIR-V to other shader languages.

Features

Convert SPIR-V to readable, usable and efficient GLSL
Convert SPIR-V to readable, usable and efficient Metal Shading Language (MSL)
Convert SPIR-V to readable, usable and efficient HLSL
Convert SPIR-V to a JSON reflection format
Convert SPIR-V to debuggable C++ [DEPRECATED]
Reflection API to simplify the creation of Vulkan pipeline layouts
Reflection API to modify and tweak OpDecorations
Supports "all" of vertex, fragment, tessellation, geometry and compute shaders.

SPIRV-Cross tries hard to emit readable and clean output from the SPIR-V. The goal is to emit GLSL or MSL that looks like it was written by a human and not awkward IR/assembly-like code.

NOTE: Individual features are expected to be mostly complete, but it is possible that certain obscure GLSL features are not yet supported. However, most missing features are expected to be "trivial" improvements at this stage.

Building

SPIRV-Cross has been tested on Linux, iOS/OSX, Windows and Android. CMake is the main build system.

Linux and macOS

Building with CMake is recommended, as it is the only build system which is tested in continuous integration. It is also the only build system which has install commands and other useful build system features.

However, you can just run make on the command line as a fallback if you only care about the CLI tool.

A non-ancient GCC (4.8+) or Clang (3.x+) compiler is required as SPIRV-Cross uses C++11 extensively.

Windows

Building with CMake is recommended, which is the only way to target MSVC. MinGW-w64 based compilation works with make as a fallback.

Android

SPIRV-Cross is only useful as a library here. Use the CMake build to link SPIRV-Cross to your project.

C++ exceptions

The make and CMake build flavors offer the option to treat exceptions as assertions. To disable exceptions for make just append SPIRV_CROSS_EXCEPTIONS_TO_ASSERTIONS=1 to the command line. For CMake append -DSPIRV_CROSS_EXCEPTIONS_TO_ASSERTIONS=ON. By default exceptions are enabled.

Static, shared and CLI

You can use -DSPIRV_CROSS_STATIC=ON/OFF -DSPIRV_CROSS_SHARED=ON/OFF -DSPIRV_CROSS_CLI=ON/OFF to control which modules are built (and installed).

Usage

Using the C++ API

The C++ API is the main API for SPIRV-Cross. For more in-depth documentation than what's provided in this README, please have a look at the Wiki. NOTE: This API is not guaranteed to be ABI-stable, and it is highly recommended to link against this API statically. The API is generally quite stable, but it can change over time, see the C API for more stability.

To perform reflection and convert to other shader languages you can use the SPIRV-Cross API. For example:

#include 
extern std::vector

load_spirv_file(); int main() { // Read SPIR-V from disk or similar. std::vector

spirv_binary = load_spirv_file(); spirv_cross::CompilerGLSL glsl(std::move(spirv_binary)); // The SPIR-V is now parsed, and we can perform reflection on it. spirv_cross::ShaderResources resources = glsl.get_shader_resources(); // Get all sampled images in the shader. for (auto &resource : resources.sampled_images) { unsigned set = glsl.get_decoration(resource.id, spv::DecorationDescriptorSet); unsigned binding = glsl.get_decoration(resource.id, spv::DecorationBinding); printf("Image %s at set = %u, binding = %u\n", resource.name.c_str(), set, binding); // Modify the decoration to prepare it for GLSL. glsl.unset_decoration(resource.id, spv::DecorationDescriptorSet); // Some arbitrary remapping if we want. glsl.set_decoration(resource.id, spv::DecorationBinding, set * 16 + binding); } // Set some options. spirv_cross::CompilerGLSL::Options options; options.version = 310; options.es = true; glsl.set_options(options); // Compile to GLSL, ready to give to GL driver. std::string source = glsl.compile(); }">
#include "spirv_glsl.hpp"
#include <vector>
#include <utility>

extern std::vector<uint32_t> load_spirv_file();

int main()
{
	// Read SPIR-V from disk or similar.
	std::vector<uint32_t> spirv_binary = load_spirv_file();

	spirv_cross::CompilerGLSL glsl(std::move(spirv_binary));

	// The SPIR-V is now parsed, and we can perform reflection on it.
	spirv_cross::ShaderResources resources = glsl.get_shader_resources();

	// Get all sampled images in the shader.
	for (auto &resource : resources.sampled_images)
	{
		unsigned set = glsl.get_decoration(resource.id, spv::DecorationDescriptorSet);
		unsigned binding = glsl.get_decoration(resource.id, spv::DecorationBinding);
		printf("Image %s at set = %u, binding = %u\n", resource.name.c_str(), set, binding);

		// Modify the decoration to prepare it for GLSL.
		glsl.unset_decoration(resource.id, spv::DecorationDescriptorSet);

		// Some arbitrary remapping if we want.
		glsl.set_decoration(resource.id, spv::DecorationBinding, set * 16 + binding);
	}

	// Set some options.
	spirv_cross::CompilerGLSL::Options options;
	options.version = 310;
	options.es = true;
	glsl.set_options(options);

	// Compile to GLSL, ready to give to GL driver.
	std::string source = glsl.compile();
}

Using the C API wrapper

To facilitate C compatibility and compatibility with foreign programming languages, a C89-compatible API wrapper is provided. Unlike the C++ API, the goal of this wrapper is to be fully stable, both API and ABI-wise. This is the only interface which is supported when building SPIRV-Cross as a shared library.

An important point of the wrapper is that all memory allocations are contained in the spvc_context. This simplifies the use of the API greatly. However, you should destroy the context as soon as reasonable, or use spvc_context_release_allocations() if you intend to reuse the spvc_context object again soon.

Most functions return a spvc_result, where SPVC_SUCCESS is the only success code. For brevity, the code below does not do any error checking.

#include <spirv_cross_c.h>

const SpvId *spirv = get_spirv_data();
size_t word_count = get_spirv_word_count();

spvc_context context = NULL;
spvc_parsed_ir ir = NULL;
spvc_compiler compiler_glsl = NULL;
spvc_compiler_options options = NULL;
spvc_resources resources = NULL;
const spvc_reflected_resource *list = NULL;
const char *result = NULL;
size_t count;
size_t i;

// Create context.
spvc_context_create(&context);

// Set debug callback.
spvc_context_set_error_callback(context, error_callback, userdata);

// Parse the SPIR-V.
spvc_context_parse_spirv(context, spirv, word_count, &ir);

// Hand it off to a compiler instance and give it ownership of the IR.
spvc_context_create_compiler(context, SPVC_BACKEND_GLSL, ir, SPVC_CAPTURE_MODE_TAKE_OWNERSHIP, &compiler_glsl);

// Do some basic reflection.
spvc_compiler_create_shader_resources(compiler_glsl, &resources);
spvc_resources_get_resource_list_for_type(resources, SPVC_RESOURCE_TYPE_UNIFORM_BUFFER, &list, &count);

for (i = 0; i < count; i++)
{
    printf("ID: %u, BaseTypeID: %u, TypeID: %u, Name: %s\n", list[i].id, list[i].base_type_id, list[i].type_id,
           list[i].name);
    printf("  Set: %u, Binding: %u\n",
           spvc_compiler_get_decoration(compiler_glsl, list[i].id, SpvDecorationDescriptorSet),
           spvc_compiler_get_decoration(compiler_glsl, list[i].id, SpvDecorationBinding));
}

// Modify options.
spvc_compiler_create_compiler_options(compiler_glsl, &options);
spvc_compiler_options_set_uint(options, SPVC_COMPILER_OPTION_GLSL_VERSION, 330);
spvc_compiler_options_set_bool(options, SPVC_COMPILER_OPTION_GLSL_ES, SPVC_FALSE);
spvc_compiler_install_compiler_options(compiler_glsl, options);

spvc_compiler_compile(compiler_glsl, &result);
printf("Cross-compiled source: %s\n", result);

// Frees all memory we allocated so far.
spvc_context_destroy(context);

Linking

CMake add_subdirectory()

This is the recommended way if you are using CMake and want to link against SPIRV-Cross statically.

Integrating SPIRV-Cross in a custom build system

To add SPIRV-Cross to your own codebase, just copy the source and header files from root directory and build the relevant .cpp files you need. Make sure to build with C++11 support, e.g. -std=c++11 in GCC and Clang. Alternatively, the Makefile generates a libspirv-cross.a static library during build that can be linked in.

Linking against SPIRV-Cross as a system library

It is possible to link against SPIRV-Cross when it is installed as a system library, which would be mostly relevant for Unix-like platforms.

pkg-config

For Unix-based systems, a pkg-config is installed for the C API, e.g.:

$ pkg-config spirv-cross-c-shared --libs --cflags
-I/usr/local/include/spirv_cross -L/usr/local/lib -lspirv-cross-c-shared

CMake

If the project is installed, it can be found with find_package(), e.g.:

cmake_minimum_required(VERSION 3.5)
set(CMAKE_C_STANDARD 99)
project(Test LANGUAGES C)

find_package(spirv_cross_c_shared)
if (spirv_cross_c_shared_FOUND)
        message(STATUS "Found SPIRV-Cross C API! :)")
else()
        message(STATUS "Could not find SPIRV-Cross C API! :(")
endif()

add_executable(test test.c)
target_link_libraries(test spirv-cross-c-shared)

test.c:

#include <spirv_cross_c.h>

int main(void)
{
        spvc_context context;
        spvc_context_create(&context);
        spvc_context_destroy(context);
}

CLI

The CLI is suitable for basic cross-compilation tasks, but it cannot support the full flexibility that the API can. Some examples below.

Creating a SPIR-V file from GLSL with glslang

glslangValidator -H -V -o test.spv test.frag

Converting a SPIR-V file to GLSL ES

glslangValidator -H -V -o test.spv shaders/comp/basic.comp
./spirv-cross --version 310 --es test.spv

Converting to desktop GLSL

glslangValidator -H -V -o test.spv shaders/comp/basic.comp
./spirv-cross --version 330 --no-es test.spv --output test.comp

Disable prettifying optimizations

glslangValidator -H -V -o test.spv shaders/comp/basic.comp
./spirv-cross --version 310 --es test.spv --output test.comp --force-temporary

Using shaders generated from C++ backend

Please see samples/cpp where some GLSL shaders are compiled to SPIR-V, decompiled to C++ and run with test data. Reading through the samples should explain how to use the C++ interface. A simple Makefile is included to build all shaders in the directory.

Implementation notes

When using SPIR-V and SPIRV-Cross as an intermediate step for cross-compiling between high level languages there are some considerations to take into account, as not all features used by one high-level language are necessarily supported natively by the target shader language. SPIRV-Cross aims to provide the tools needed to handle these scenarios in a clean and robust way, but some manual action is required to maintain compatibility.

HLSL source to GLSL

HLSL entry points

When using SPIR-V shaders compiled from HLSL, there are some extra things you need to take care of. First make sure that the entry point is used correctly. If you forget to set the entry point correctly in glslangValidator (-e MyFancyEntryPoint), you will likely encounter this error message:

Cannot end a function before ending the current block.
Likely cause: If this SPIR-V was created from glslang HLSL, make sure the entry point is valid.

Vertex/Fragment interface linking

HLSL relies on semantics in order to effectively link together shader stages. In the SPIR-V generated by glslang, the transformation from HLSL to GLSL ends up looking like

struct VSOutput {
   // SV_Position is rerouted to gl_Position
   float4 position : SV_Position;
   float4 coord : TEXCOORD0;
};

VSOutput main(...) {}

struct VSOutput {
   float4 coord;
}
layout(location = 0) out VSOutput _magicNameGeneratedByGlslang;

While this works, be aware of the type of the struct which is used in the vertex stage and the fragment stage. There may be issues if the structure type name differs in vertex stage and fragment stage.

You can make use of the reflection interface to force the name of the struct type.

// Something like this for both vertex outputs and fragment inputs.
compiler.set_name(varying_resource.base_type_id, "VertexFragmentLinkage");

Some platform may require identical variable name for both vertex outputs and fragment inputs. (for example MacOSX) to rename variable base on location, please add

--rename-interface-variable

HLSL source to legacy GLSL/ESSL

HLSL tends to emit varying struct types to pass data between vertex and fragment. This is not supported in legacy GL/GLES targets, so to support this, varying structs are flattened. This is done automatically, but the API user might need to be aware that this is happening in order to support all cases.

Modern GLES code like this:

struct Output {
   vec4 a;
   vec2 b;
};
out Output vout;

Is transformed into:

struct Output {
   vec4 a;
   vec2 b;
};
varying vec4 Output_a;
varying vec2 Output_b;

Note that now, both the struct name and the member names will participate in the linking interface between vertex and fragment, so API users might want to ensure that both the struct names and member names match so that vertex outputs and fragment inputs can link properly.

Separate image samplers (HLSL/Vulkan) for backends which do not support it (GLSL)

Another thing you need to remember is when using samplers and textures in HLSL these are separable, and not directly compatible with GLSL. If you need to use this with desktop GL/GLES, you need to call Compiler::build_combined_image_samplers first before calling Compiler::compile, or you will get an exception.

get_name(remap.image_id), compiler->get_name(remap.sampler_id))); }">

// From main.cpp
// Builds a mapping for all combinations of images and samplers.
compiler->build_combined_image_samplers();

// Give the remapped combined samplers new names.
// Here you can also set up decorations if you want (binding = #N).
for (auto &remap : compiler->get_combined_image_samplers())
{
   compiler->set_name(remap.combined_id, join("SPIRV_Cross_Combined", compiler->get_name(remap.image_id),
            compiler->get_name(remap.sampler_id)));
}

If your target is Vulkan GLSL, --vulkan-semantics will emit separate image samplers as you'd expect. The command line client calls Compiler::build_combined_image_samplers automatically, but if you're calling the library, you'll need to do this yourself.

Descriptor sets (Vulkan GLSL) for backends which do not support them (pre HLSL 5.1 / GLSL)

Descriptor sets are unique to Vulkan, so make sure that descriptor set + binding is remapped to a flat binding scheme (set always 0), so that other APIs can make sense of the bindings. This can be done with Compiler::set_decoration(id, spv::DecorationDescriptorSet). For other backends like MSL and HLSL, descriptor sets can be used, with some minor caveats, see below.

MSL 2.0+

Metal supports indirect argument buffers (--msl-argument-buffers). In this case, descriptor sets become argument buffers, and bindings are mapped to [[id(N)]] within the argument buffer. One quirk is that arrays of resources consume multiple ids, where Vulkan does not. This can be worked around either from shader authoring stage or remapping bindings as needed to avoid the overlap. There is also a rich API to declare remapping schemes which is intended to work like the pipeline layout in Vulkan. See CompilerMSL::add_msl_resource_binding. Remapping combined image samplers for example must be split into two bindings in MSL, so it's possible to declare an id for the texture and sampler binding separately.

HLSL - SM 5.1+

In SM 5.1+, descriptor set bindings are interpreted as register spaces directly. In HLSL however, arrays of resources consume multiple binding slots where Vulkan does not, so there might be overlap if the SPIR-V was not authored with this in mind. This can be worked around either from shader authoring stage (don't assign overlapping bindings) or remap bindings in SPIRV-Cross as needed to avoid the overlap.

Linking by name for targets which do not support explicit locations (legacy GLSL/ESSL)

Modern GLSL and HLSL sources (and SPIR-V) relies on explicit layout(location) qualifiers to guide the linking process between shader stages, but older GLSL relies on symbol names to perform the linking. When emitting shaders with older versions, these layout statements will be removed, so it is important that the API user ensures that the names of I/O variables are sanitized so that linking will work properly. The reflection API can rename variables, struct types and struct members to deal with these scenarios using Compiler::set_name and friends.

Clip-space conventions

SPIRV-Cross can perform some common clip space conversions on gl_Position/SV_Position by enabling CompilerGLSL::Options.vertex.fixup_clipspace. While this can be convenient, it is recommended to modify the projection matrices instead as that can achieve the same result.

For GLSL targets, enabling this will convert a shader which assumes [0, w] depth range (Vulkan / D3D / Metal) into [-w, w] range. For MSL and HLSL targets, enabling this will convert a shader in [-w, w] depth range (OpenGL) to [0, w] depth range.

By default, the CLI will not enable fixup_clipspace, but in the API you might want to set an explicit value using CompilerGLSL::set_options().

Y-flipping of gl_Position and similar is also supported. The use of this is discouraged, because relying on vertex shader Y-flipping tends to get quite messy. To enable this, set CompilerGLSL::Options.vertex.flip_vert_y or --flip-vert-y in CLI.

Reserved identifiers

When cross-compiling, certain identifiers are considered to be reserved by the implementation. Code generated by SPIRV-Cross cannot emit these identifiers as they are reserved and used for various internal purposes, and such variables will typically show up as _RESERVED_IDENTIFIER_FIXUP_ or some similar name to make it more obvious that an identifier has been renamed.

Reflection output will follow the exact name specified in the SPIR-V module. It might not be a valid identifier in the C sense, as it may contain non-alphanumeric/non-underscore characters.

Reserved identifiers currently assumed by the implementation are (in pseudo-regex):

_$digit+, e.g. _100, _2
$digit+.+, e.g. _100_tmp, _2_foobar. _2Bar is not reserved.
gl_- prefix
spv- prefix
SPIRV_Cross prefix. This prefix is generally used for interface variables where app needs to provide data for workaround purposes. This identifier will not be rewritten, but be aware of potential collisions.
Double underscores (reserved by all target languages).

Members of structs also have a reserved identifier:

_m$digit+$END, e.g. _m20 and _m40 are reserved, but not _m40Foobar.

Contributing

Contributions to SPIRV-Cross are welcome. See Testing and Licensing sections for details.

Testing

SPIRV-Cross maintains a test suite of shaders with reference output of how the output looks after going through a roundtrip through glslangValidator/spirv-as then back through SPIRV-Cross again. The reference files are stored inside the repository in order to be able to track regressions.

All pull requests should ensure that test output does not change unexpectedly. This can be tested with:

./checkout_glslang_spirv_tools.sh # Checks out glslang and SPIRV-Tools at a fixed revision which matches the reference output.
./build_glslang_spirv_tools.sh    # Builds glslang and SPIRV-Tools.
./test_shaders.sh                 # Runs over all changes and makes sure that there are no deltas compared to reference files.

./test_shaders.sh currently requires a Makefile setup with GCC/Clang to be set up. However, on Windows, this can be rather inconvenient if a MinGW environment is not set up. To use a spirv-cross binary you built with CMake (or otherwise), you can pass in an environment variable as such:

SPIRV_CROSS_PATH=path/to/custom/spirv-cross ./test_shaders.sh

However, when improving SPIRV-Cross there are of course legitimate cases where reference output should change. In these cases, run:

./update_test_shaders.sh          # SPIRV_CROSS_PATH also works here.

to update the reference files and include these changes as part of the pull request. Always make sure you are running the correct version of glslangValidator as well as SPIRV-Tools when updating reference files. See checkout_glslang_spirv_tools.sh which revisions are currently expected. The revisions change regularly.

In short, the master branch should always be able to run ./test_shaders.py shaders and friends without failure. SPIRV-Cross uses Travis CI to test all pull requests, so it is not strictly needed to perform testing yourself if you have problems running it locally. A pull request which does not pass testing on Travis will not be accepted however.

When adding support for new features to SPIRV-Cross, a new shader and reference file should be added which covers usage of the new shader features in question. Travis CI runs the test suite with the CMake, by running ctest. This is a more straight-forward alternative to ./test_shaders.sh.

Licensing

Contributors of new files should add a copyright header at the top of every new source code file with their copyright along with the Apache 2.0 licensing stub.

Formatting

SPIRV-Cross uses clang-format to automatically format code. Please use clang-format with the style sheet found in .clang-format to automatically format code before submitting a pull request.

To make things easy, the format_all.sh script can be used to format all source files in the library. In this directory, run the following from the command line:

./format_all.sh

Regression testing

In shaders/ a collection of shaders are maintained for purposes of regression testing. The current reference output is contained in reference/. ./test_shaders.py shaders can be run to perform regression testing.

See ./test_shaders.py --help for more.

Metal backend

To test the roundtrip path GLSL -> SPIR-V -> MSL, --msl can be added, e.g. ./test_shaders.py --msl shaders-msl.

HLSL backend

To test the roundtrip path GLSL -> SPIR-V -> HLSL, --hlsl can be added, e.g. ./test_shaders.py --hlsl shaders-hlsl.

Updating regression tests

When legitimate changes are found, use --update flag to update regression files. Otherwise, ./test_shaders.py will fail with error code.

Mali Offline Compiler cycle counts

To obtain a CSV of static shader cycle counts before and after going through spirv-cross, add --malisc flag to ./test_shaders. This requires the Mali Offline Compiler to be installed in PATH.

Comments

MSL: gl_Position is undetectable in tessellation stage IO using the SPIRV-Cross reflection API
At Unity we're working on adding Metal tessellation support using SPIRV-Cross but have hit a limitation, hopefully there's already a way of doing this that we're unaware of.

The issue is that there does not seem to be a way to tell when and at what offset the generated MSL will contain a gl_Position member in IO interface for the Vertex or Hull kernels or the Domain vertex function as it's not reported along with the other SPVC_RESOURCE_TYPE_STAGE_INPUT or SPVC_RESOURCE_TYPE_STAGE_OUTPUT resource list members even though it's in the generated MSL.

This results in the following problems when trying to validate and link up the different Metal stages:

To be able to set up control-point attribute data (MTLVertexAttributeDescriptor data in MTLVertexDescriptor ) for the post-tess vertex (domain) stage it is necessary to know the layout of the Hull kernel's control point output buffer. As the data transfer is done via buffer it does not seem possible to work around this and get the data layout from the Metal reflection for the Domain MTLFunction.

If we don't know that gl_Position is in the vertex kernel output or the hull kernel output, we cannot know the control point size/stride to be able to allocate minimally sized buffers for communication between the tessellation stages (this can be worked around by always assuming gl_Position is there and padding each point with a float4 though is suboptimal)

As we use --msl-multi-patch-workgroup the communication between the vertex and hull kernels happens directly through a buffer. We'd like to validate that the format is identical but since we cannot know if only one of them contains gl_Position this can't be done for all cases. Note that if --msl-multi-patch-workgroup is not used, stage IO validation is still not possible since Vertex stage output may or may not contain gl_Position even though it could be detected in the Hull stage input using the Metal runtime reflection (but not using SPIRV-Cross)

Similar as point 3, we'd like to be able to verify that the gl_Position post-tess vertex (domain) function input attribute was written to by the previous stage after compiling the two stages with SPIRV-Cross

Currently the only solution seems to be to manually parse out the OpEntryPoint instruction and to go over the list of Interface <id>s and only use SPIRV-Cross to check if any of them have a BuiltIn decoration of Position. This also requires us to assume that the gl_Position member will always be places at the end of the IO structs, as there's no way to get its offset in the IO buffers as well. As the placement looks like an arbitrary SPIRV-Cross implementation detail it does not seem like something we should rely on.

The following is the source for the vert/hull/domain stages: tess_source.txt

This is the associated SPIR-V after compiling with DXC (binaries downloaded from shader playground, .o files renamed to .spv.txt to upload to github) Vert.spv.txt Hull.spv.txt Domain.spv.txt

The following is the MSL generated with SPIRV-Cross Vert.msl.txt ./spirv-cross --msl --msl-vertex-for-tessellation Vert.spv.txt --output Vert.msl.txt Hull.msl.txt ./spirv-cross --msl --msl-multi-patch-workgroup Hull.spv.txt --output Hull.msl.txt Domain.msl.txt ./spirv-cross --msl Domain.spv.txt --output Domain.msl.txt

question
opened by Dredhog 51

MSL : Translate C style array to array causes regression on Dawn end2end tests

After applying this patch, VertexFormatTests) from end2end tests of Dawn is broken on Mac of Intel Iris Pro platform. The root cause seems to be Intel platform doesn't support variable as indexing of array<array> type on Iris Pro. Here are detail infos. This is the original glsl shader :

#version 450
        layout(location = 0) in ivec2 test;
        layout(location = 0) out vec4 color;
        
        void main() {
            int expected[3][2];
            expected[0][0] = int(127);
            expected[0][1] = int(0);
            expected[1][0] = int(-128);
            expected[1][1] = int(-2);
            expected[2][0] = int(120);
            expected[2][1] = int(-121);
            
            bool success = true;
            int testVal0;
            int expectedVal0;
            int testVal1;
            int expectedVal1;
            testVal0 = test[0];
            testVal1 = test[1];
            expectedVal0 = expected[gl_VertexIndex][0];
            expectedVal1 = expected[gl_VertexIndex][1];
            /* Workaround     
                if (gl_VertexIndex == 0) {
                    expectedVal0 = expected[0][0];
                    expectedVal1 = expected[0][1];
                } else if (gl_VertexIndex == 1) {
                    expectedVal0 = expected[1][0];
                    expectedVal1 = expected[1][1];
                } else {
                    expectedVal0 = expected[2][0];
                    expectedVal1 = expected[2][1];
                }
            }*/
            success = success && (testVal0 == expectedVal0);
            success = success && (testVal1 == expectedVal1);
            if (success) {
                color = vec4(0.0f, 1.0f, 0.0f, 1.0f);
            } else {
                color = vec4(1.0f, 0.0f, 0.0f, 1.0f);
            }
           
            const vec2 pos[3] = vec2[3](vec2(0.0f, 0.5f), vec2(-0.5f, -0.5f), vec2(0.5f, -0.5f));
            gl_Position = vec4(pos[gl_VertexIndex], 0.0, 1.0);

And the translated metal shader before applying array patch is :

#include <simd/simd.h>

using namespace metal;

constant float2 _88[3] = { float2(0.0, 0.5), float2(-0.5), float2(0.5, -0.5) };

struct main0_out
{
    float4 color [[user(locn0)]];
    float4 gl_Position [[position]];
};

struct main0_in
{
    int2 test [[attribute(0)]];
};

vertex main0_out main0(main0_in in [[stage_in]], uint gl_VertexIndex [[vertex_id]])
{
    main0_out out = {};
    int expected[3][2];
    expected[0][0] = 127;
    expected[0][1] = 0;
    expected[1][0] = -128;
    expected[1][1] = -2;
    expected[2][0] = 120;
    expected[2][1] = -121;
    bool success = true;
    bool useVariable = true;
    int testVal0 = in.test.x;
    int testVal1 = in.test.y;
    int expectedVal0 = expected[int(gl_VertexIndex)][0];
    int expectedVal1 = expected[int(gl_VertexIndex)][1];
    success = success && (testVal0 == expectedVal0);
    success = success && (testVal1 == expectedVal1);
    if (success)
    {
        out.color = float4(0.0, 1.0, 0.0, 1.0);
    }
    else
    {
        out.color = float4(1.0, 0.0, 0.0, 1.0);
    }
    out.gl_Position = float4(_88[int(gl_VertexIndex)], 0.0, 1.0);
    return out;
}

The translated metal shader after applying array patch is :

#pragma clang diagnostic ignored "-Wmissing-braces"
#pragma clang diagnostic ignored "-Wunused-variable"

#include <metal_stdlib>
#include <simd/simd.h>
	
template <typename T, size_t Num>
struct unsafe_array
{
	T __Elements[Num ? Num : 1];
	
	constexpr size_t size() const thread { return Num; }
	constexpr size_t max_size() const thread { return Num; }
	constexpr bool empty() const thread { return Num == 0; }
	
	constexpr size_t size() const device { return Num; }
	constexpr size_t max_size() const device { return Num; }
	constexpr bool empty() const device { return Num == 0; }
	
	constexpr size_t size() const constant { return Num; }
	constexpr size_t max_size() const constant { return Num; }
	constexpr bool empty() const constant { return Num == 0; }
	
	constexpr size_t size() const threadgroup { return Num; }
	constexpr size_t max_size() const threadgroup { return Num; }
	constexpr bool empty() const threadgroup { return Num == 0; }
	
	thread T &operator[](size_t pos) thread
	{
		return __Elements[pos];
	}
	constexpr const thread T &operator[](size_t pos) const thread
	{
		return __Elements[pos];
	}
	
	device T &operator[](size_t pos) device
	{
		return __Elements[pos];
	}
	constexpr const device T &operator[](size_t pos) const device
	{
		return __Elements[pos];
	}
	
	constexpr const constant T &operator[](size_t pos) const constant
	{
		return __Elements[pos];
	}
	
	threadgroup T &operator[](size_t pos) threadgroup
	{
		return __Elements[pos];
	}
	constexpr const threadgroup T &operator[](size_t pos) const threadgroup
	{
		return __Elements[pos];
	}
};

using namespace metal;

constant unsafe_array<float2,3> _88 = unsafe_array<float2,3>({ float2(0.0, 0.5), float2(-0.5), float2(0.5, -0.5) });

struct main0_out
{
    float4 color [[user(locn0)]];
    float4 gl_Position [[position]];
};

struct main0_in
{
    int2 test [[attribute(0)]];
};

vertex main0_out main0(main0_in in [[stage_in]], uint gl_VertexIndex [[vertex_id]])
{
    main0_out out = {};
    unsafe_array<unsafe_array<int,3>,2> expected;
    expected[0][0] = 127;
    expected[0][1] = 0;
    expected[1][0] = -128;
    expected[1][1] = -2;
    expected[2][0] = 120;
    expected[2][1] = -121;
    bool success = true;
    bool useVariable = true;
    int testVal0 = in.test.x;
    int testVal1 = in.test.y;
    int expectedVal0 = expected[int(gl_VertexIndex)][0];
    int expectedVal1 = expected[int(gl_VertexIndex)][1];
    success = success && (testVal0 == expectedVal0);
    success = success && (testVal1 == expectedVal1);
    if (success)
    {
        out.color = float4(0.0, 1.0, 0.0, 1.0);
    }
    else
    {
        out.color = float4(1.0, 0.0, 0.0, 1.0);
    }
    out.gl_Position = float4(_88[int(gl_VertexIndex)], 0.0, 1.0);
    return out;
}

The only difference between these two shaders are using of unsafe_array to replace C style array.

I'm doing some initial experiment and find a simple workaround in glsl is to replace

            expectedVal0 = expected[gl_VertexIndex][0];
            expectedVal1 = expected[gl_VertexIndex][1];

with a non variable indexing form like this:

 if (gl_VertexIndex == 0) {
                    expectedVal0 = expected[0][0];
                    expectedVal1 = expected[0][1];
                } else if (gl_VertexIndex == 1) {
                    expectedVal0 = expected[1][0];
                    expectedVal1 = expected[1][1];
                } else {
                    expectedVal0 = expected[2][0];
                    expectedVal1 = expected[2][1];
                }

can make the tests work again. This proves that Intel platform doesn't support variable as indexing of array<array> type on Iris Pro.

I also did some experiment on MacOS of UHD630 Intel graphics platform, it turns out that all the cases using variable as indexing of a 1D array failed.

I think this maybe a driver issue for Intel GPU on metal(Need to double check). And for spirv-cross, can we workaround Intel Platform by not translating C style array to array form?

question needs triage workaround

opened by shaoboyan 46

Segmentation fault when initializing CompilerGLSL object.

I am trying to parse some spirv code generated from glsl and it crashes on construction. I am sure the spirv array is correct because I am creating vulkan shader modules using shaderc that compile and run fine.

This is the smallest program I could make that reproduces the error:


   Compiler compiler;
   CompileOptions options;
   shaderc::SpvCompilationResult compilation_result = compiler.CompileGlslToSpv(
       shader_source, shaderc_shader_kind::shaderc_glsl_vertex_shader, shader_path.c_str(), options);

   std::vector<uint32_t> spirv_shader;
   spirv_shader.assign(compilation_result.cbegin(), compilation_result.cend());

   std::vector<uint32_t> source_copy = spirv_shader;
   spirv_cross::CompilerGLSL glsl(source_copy);

where shader_source is the std string containing the glsl string of the shader.

With input:

#version 450

layout(location = 0) in vec2 inPosition;
layout(location = 1) in vec3 inColor;

layout(location = 0) out vec3 fragColor;

layout(binding = 0) uniform MatUBO
{
   mat2 transform;
};


void main() {
   gl_Position = vec4(inPosition, 0.5, 1.0);
   fragColor = inColor;
}

OS Ubuntu 21.10, compiler clang. The error is a segmentation fault thrown at line: spirv_common.hpp:86

opened by Makogan 29

Lots of changes and fixes with new test cases to make SPIRV-Cross work in UE4 for Metal

This pull request is the result of a bunch of work from my former colleague and some adjustments by myself in order to make SPIRV-Cross work in conjunction with ShaderConductor in Unreal Engine 4.

Several new test cases were added to justify the changes that were made. I also renamed the respective test shaders to match the guideline of this project. Most of the changed files in this pull request are the updated reference shaders.

One of the most notable changes is the introduction of the spvUnsafeArray<> template struct to make arrays a value type in Metal. It's called "unsafe" because there are no boundary checks and it's really just a wrapper with respective operator [] functions around a native array to make them compatible with how structured and byte-address buffers work. The spvArrayCopy* functions can now be avoided most of the time to let the Metal compiler do a better optimization. However, spvArrayCopy* wrappers are still required when arrays are declared in threadgroup address space.

Another change worth mentioning is that the translation of HLSL's SV_InstanceID and SV_VertexID works correct now, since these system values always start at 0. Metal's [[instance_id]]/[[vertex_id]] and GLSL's gl_VertexIndex/gl_InstanceIndex, however, always start at the base index specified in the draw command. Both Metal and GLSL support system values for these base indices which are now exposed to make the translation of SV_InstanceID and SV_VertexID behave in the same way.

We also had to re-enable some features that were removed recently, like the user defined decoration binding, which can now be enabled again with the respective boolean in CompilerMSL::Options but it's disabled by default, to keep the behavior as it was without these changes.

We hope to get these changes into Khronos' repository to simplify our update iterations. Any feedback is highly welcome.

opened by LukasBanana 27
Treat array of writable textures as individual textures when cross-compiling to MSL
Hello! We recently faced a problem with array of writable textures in MSL: the following shader fails to correctly compile after conversion to MSL (https://github.com/KhronosGroup/MoltenVK/issues/517, https://github.com/DiligentGraphics/DiligentCore/issues/66):

layout(rgba8) uniform writeonly image2DArray OutMip[4]; ... imageStore(OutMip[3], ivec3(GlobalInd.xy / 8u, ArraySlice), PackColor(Src1));

The problem is that it does not seem that array of textures can be writable in MSL. @billhollings suggests that this can potentially be remedied by by forcing an array of writable textures to be treated as individual textures and assigning consecutive Metal texture indices to the declarations.

Is that something that you may consider to implement?
wontfix
opened by DiligentGraphics 27

Crash in spirv_cross::Meta::~Meta() on iOS

Hello!

I am seeing the following crash when releasing spirv-cross compiler:

#0	0x000000010445a308 in std::__1::__hash_table<std::__1::__hash_value_type<unsigned int, unsigned int>, std::__1::__unordered_map_hasher<unsigned int, std::__1::__hash_value_type<unsigned int, unsigned int>, std::__1::hash<unsigned int>, true>, std::__1::__unordered_map_equal<unsigned int, std::__1::__hash_value_type<unsigned int, unsigned int>, std::__1::equal_to<unsigned int>, true>, std::__1::allocator<std::__1::__hash_value_type<unsigned int, unsigned int> > >::__deallocate_node(std::__1::__hash_node_base<std::__1::__hash_node<std::__1::__hash_value_type<unsigned int, unsigned int>, void*>*>*) at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__hash_table:1547
#1	0x000000010445a1bc in std::__1::__hash_table<std::__1::__hash_value_type<unsigned int, unsigned int>, std::__1::__unordered_map_hasher<unsigned int, std::__1::__hash_value_type<unsigned int, unsigned int>, std::__1::hash<unsigned int>, true>, std::__1::__unordered_map_equal<unsigned int, std::__1::__hash_value_type<unsigned int, unsigned int>, std::__1::equal_to<unsigned int>, true>, std::__1::allocator<std::__1::__hash_value_type<unsigned int, unsigned int> > >::~__hash_table() at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__hash_table:1503
#2	0x000000010445a16c in std::__1::__hash_table<std::__1::__hash_value_type<unsigned int, unsigned int>, std::__1::__unordered_map_hasher<unsigned int, std::__1::__hash_value_type<unsigned int, unsigned int>, std::__1::hash<unsigned int>, true>, std::__1::__unordered_map_equal<unsigned int, std::__1::__hash_value_type<unsigned int, unsigned int>, std::__1::equal_to<unsigned int>, true>, std::__1::allocator<std::__1::__hash_value_type<unsigned int, unsigned int> > >::~__hash_table() at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__hash_table:1495
#3	0x000000010445a138 in std::__1::unordered_map<unsigned int, unsigned int, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, unsigned int> > >::~unordered_map() at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__hash_table:1370
#4	0x0000000104459968 in std::__1::unordered_map<unsigned int, unsigned int, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, unsigned int> > >::~unordered_map() at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__hash_table:1370
#5	0x000000010445a900 in spirv_cross::Meta::~Meta() at /git/DiligentEngine/DiligentCore/External/SPIRV-Cross/spirv_common.hpp:1381
#6	0x000000010445a8c4 in spirv_cross::Meta::~Meta() at /git/DiligentEngine/DiligentCore/External/SPIRV-Cross/spirv_common.hpp:1381
#7	0x000000010445a890 in std::__1::pair<unsigned int const, spirv_cross::Meta>::~pair() at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/utility:312
#8	0x000000010445a854 in std::__1::pair<unsigned int const, spirv_cross::Meta>::~pair() at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/utility:312
#9	0x000000010445a7d8 in void std::__1::allocator_traits<std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<unsigned int, spirv_cross::Meta>, void*> > >::__destroy<std::__1::pair<unsigned int const, spirv_cross::Meta> >(std::__1::integral_constant<bool, false>, std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<unsigned int, spirv_cross::Meta>, void*> >&, std::__1::pair<unsigned int const, spirv_cross::Meta>*) [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:1732
#10	0x000000010445a7d0 in void std::__1::allocator_traits<std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<unsigned int, spirv_cross::Meta>, void*> > >::destroy<std::__1::pair<unsigned int const, spirv_cross::Meta> >(std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<unsigned int, spirv_cross::Meta>, void*> >&, std::__1::pair<unsigned int const, spirv_cross::Meta>*) [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:1595
#11	0x000000010445a7c0 in std::__1::__hash_table<std::__1::__hash_value_type<unsigned int, spirv_cross::Meta>, std::__1::__unordered_map_hasher<unsigned int, std::__1::__hash_value_type<unsigned int, spirv_cross::Meta>, std::__1::hash<unsigned int>, true>, std::__1::__unordered_map_equal<unsigned int, std::__1::__hash_value_type<unsigned int, spirv_cross::Meta>, std::__1::equal_to<unsigned int>, true>, std::__1::allocator<std::__1::__hash_value_type<unsigned int, spirv_cross::Meta> > >::__deallocate_node(std::__1::__hash_node_base<std::__1::__hash_node<std::__1::__hash_value_type<unsigned int, spirv_cross::Meta>, void*>*>*) at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__hash_table:1564
#12	0x000000010445a614 in std::__1::__hash_table<std::__1::__hash_value_type<unsigned int, spirv_cross::Meta>, std::__1::__unordered_map_hasher<unsigned int, std::__1::__hash_value_type<unsigned int, spirv_cross::Meta>, std::__1::hash<unsigned int>, true>, std::__1::__unordered_map_equal<unsigned int, std::__1::__hash_value_type<unsigned int, spirv_cross::Meta>, std::__1::equal_to<unsigned int>, true>, std::__1::allocator<std::__1::__hash_value_type<unsigned int, spirv_cross::Meta> > >::~__hash_table() at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__hash_table:1503
#13	0x000000010445a5c4 in std::__1::__hash_table<std::__1::__hash_value_type<unsigned int, spirv_cross::Meta>, std::__1::__unordered_map_hasher<unsigned int, std::__1::__hash_value_type<unsigned int, spirv_cross::Meta>, std::__1::hash<unsigned int>, true>, std::__1::__unordered_map_equal<unsigned int, std::__1::__hash_value_type<unsigned int, spirv_cross::Meta>, std::__1::equal_to<unsigned int>, true>, std::__1::allocator<std::__1::__hash_value_type<unsigned int, spirv_cross::Meta> > >::~__hash_table() at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__hash_table:1495
#14	0x000000010445a590 in std::__1::unordered_map<unsigned int, spirv_cross::Meta, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, spirv_cross::Meta> > >::~unordered_map() at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__hash_table:1370
#15	0x00000001044599d0 in std::__1::unordered_map<unsigned int, spirv_cross::Meta, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, spirv_cross::Meta> > >::~unordered_map() at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__hash_table:1370
#16	0x00000001044598a8 in spirv_cross::ParsedIR::~ParsedIR() at /git/DiligentEngine/DiligentCore/External/SPIRV-Cross/spirv_cross_parsed_ir.hpp:33
#17	0x0000000104da9a70 in spirv_cross::Compiler::~Compiler() ()
#18	0x0000000104ddc920 in spirv_cross::Compiler::~Compiler() ()

There is nothing special about how compiler is used as well as about the byte code it parses:

std::unique_ptr<spirv_cross::Parser> pParser(new spirv_cross::Parser(std::move(spirv_binary)));
pParser->parse();
auto ParsedIRSource = pParser->get_parsed_ir().source;
std::unique_ptr<spirv_cross::Compiler> pCompiler(new spirv_cross::Compiler(std::move(pParser->get_parsed_ir())));

This code works fine on Windows, Linux and Mac, but on iOS it crashes. If both parser and compiler are not destroyed, the app continues successfully and runs as expected.

ready for work cross-project

opened by DiligentGraphics 23

GLSL: RelaxedPrecision decoration applied to operation result IDs is not correctly translated to essl

Hi, It appears that SPIRV-Cross might not be translating RelaxedPrecision decorations correctly into essl.

This is similar to case https://github.com/KhronosGroup/SPIRV-Cross/issues/1855 but more general.

Here's an example HLSL fragment shader:

min16float RelaxedCalc(min16float3 input)
{
    min16float temp0 = input.x;
    min16float temp1 = input.y;
    min16float temp2 = input.z;
    return temp0 * temp1 * temp2;
}

float FloatCalc(min16float3 input)
{
    float temp0 = input.x;
    float temp1 = input.y;
    float temp2 = input.z;
    return temp0 * temp1 * temp2;
}

float4 PSMain(min16float4 color : COLOR) : SV_TARGET
{
    return float4(RelaxedCalc(color), FloatCalc(color), 0, 0);
}

Here's the SPIR-V that DXC generates for it:

; SPIR-V
; Version: 1.0
; Generator: Google spiregg; 0
; Bound: 21
; Schema: 0
               OpCapability Shader
               OpMemoryModel Logical GLSL450
               OpEntryPoint Fragment %PSMain "PSMain" %in_var_COLOR %out_var_SV_TARGET
               OpExecutionMode %PSMain OriginUpperLeft
               OpSource HLSL 600
               OpName %in_var_COLOR "in.var.COLOR"
               OpName %out_var_SV_TARGET "out.var.SV_TARGET"
               OpName %PSMain "PSMain"
               OpDecorate %in_var_COLOR Location 0
               OpDecorate %out_var_SV_TARGET Location 0
               OpDecorate %in_var_COLOR RelaxedPrecision
               OpDecorate %4 RelaxedPrecision
               OpDecorate %5 RelaxedPrecision
               OpDecorate %6 RelaxedPrecision
               OpDecorate %7 RelaxedPrecision
               OpDecorate %8 RelaxedPrecision
               OpDecorate %9 RelaxedPrecision
      %float = OpTypeFloat 32
    %float_0 = OpConstant %float 0
    %v4float = OpTypeVector %float 4
%_ptr_Input_v4float = OpTypePointer Input %v4float
%_ptr_Output_v4float = OpTypePointer Output %v4float
       %void = OpTypeVoid
         %16 = OpTypeFunction %void
%in_var_COLOR = OpVariable %_ptr_Input_v4float Input
%out_var_SV_TARGET = OpVariable %_ptr_Output_v4float Output
     %PSMain = OpFunction %void None %16
         %17 = OpLabel
          %4 = OpLoad %v4float %in_var_COLOR
          %5 = OpCompositeExtract %float %4 0
          %6 = OpCompositeExtract %float %4 1
          %7 = OpCompositeExtract %float %4 2
          %8 = OpFMul %float %5 %6
          %9 = OpFMul %float %8 %7
         %18 = OpFMul %float %5 %6
         %19 = OpFMul %float %18 %7
         %20 = OpCompositeConstruct %v4float %9 %19 %float_0 %float_0
               OpStore %out_var_SV_TARGET %20
               OpReturn
               OpFunctionEnd

SPIRV-Cross generates the following essl:

#version 320 es
precision mediump float;
precision highp int;

layout(location = 0) in vec4 in_var_COLOR;
layout(location = 0) out highp vec4 out_var_SV_TARGET;

void main()
{
    out_var_SV_TARGET = vec4((in_var_COLOR.x * in_var_COLOR.y) * in_var_COLOR.z, (in_var_COLOR.x * in_var_COLOR.y) * in_var_COLOR.z, 0.0, 0.0);
}

Basically, in the essl produced by SPIRV-Cross, all of the multiplications are allowed to run at mediump, even though only the result IDs of the multiplications in RelaxedCalc i.e. %8, %9 are decorated with RelaxedPrecision, and the result IDs corresponding to FloatCalc i.e. %19, %20 don't have RelaxedPrecision decorations, which to my understanding is not correct.

What I would expect is that whenever an operation doesn't have a RelaxedPrecision decoration the mediump inputs for it in the essl should first be moved into highp temporaries (only necessary when there isn't at least one highp input already). The opposite should also be true i.e. whenever an operation is decorated with RelaxedPrecision all of the operands which aren't already mediump should be first moved into mediump temporaries.

Here's my understanding of the situation:

In SPIR-V decorating the result of e.g. an OpFMul with RelaxedPrecision means it should execute at relaxed precision.
Not decorating OpFMul with RelaxedPrecision means it should execute at full precision, this is independent of if the operands have RelaxedPrecision decorations.
For RelaxedPrecision variables the only operations which implicitly inherit the decoration seem to be loads and stores, not arithmetic ops, this is how I understand the SPIR-V spec.
On the other hand, in essl, an operation's precision is independent of the destination's precision but depends on the precision of the operands (I believe there are some rare exceptions when none of the operands have a precision specified, but I don't think this applies to SPIRV-Cross as it always provides a default precision).

enhancement in progress

opened by Dredhog 22

Metal indirect argument buffers
Feature

SPIRV-Cross needs to be able to generate argument buffer descriptions from given bindings. For example, given the SPIRV generated from the following GLSL:

layout(set = 0, binding = 0) uniform texture2D u_Texture; layout(set = 0, binding = 1) uniform sampler u_Sampler;

The Metal-1 version of the PS entry point (using those bindings) would look like:

fragment float4 ps_main(VsOutput in [[stage_in]], texture2d<float> u_Texture [[ texture(0) ]], sampler u_Sampler [[ sampler(1) ]]) {...}

With Metal-2 and the indirect argument buffers, we'd need the following code generated:

struct IndirectArgSet0 { // contains all the resources with `set = 0` texture2d<float> u_Texture [[ id(0) ]]; // binding indices are the same sampler u_Sampler [[ id(1) ]]; }; fragment float4 ps_main(VsOutput in [[stage_in]], device IndirectArgSet0* pixelArg [[ buffer(0) ]]) // note the buffer index {...}

Using indirect argument buffers provides a fast path of binding resources on Metal2, similar to how descriptor sets are bound in Vulkan/DX12.

Issues

One implementation detail that is not clear is what buffer index to assign to. A vertex program may still have vertex attributes provided in buffers (while everything else will go into the indirect arguments), so a collision is possible. I propose an option to specify the starting indirect buffer binding:

spirv-cross --msl-indirect-argument-buffers <start_index>

Background

gfx-rs has the indirect argument code path implemented for the descriptor set binding implementation on Metal2. Here is the switch in the core/quad example. We currently provide MSL code manually, which is obviously different between regular and indirect argument paths. We'd like both versions to be generated by SPIRV-Cross.
enhancement in progress
opened by kvark 20

MSL: RWBuffer is translated into a texture2d and buffer of atomic_int even though SPIRV-Cross can tell the original type

We're using DXC to turn the following HLSL:

RWBuffer<uint> g_DispatchIndirectBuffer : register( u0 );

[numthreads(1, 1, 1)]
void CSMain()
{
    uint prevGroupCnt;
    InterlockedAdd(g_DispatchIndirectBuffer[0], 4, prevGroupCnt);
}

Into the following SPIR-V

; SPIR-V
; Version: 1.0
; Generator: Google spiregg; 0
; Bound: 15
; Schema: 0
               OpCapability Shader
               OpCapability SampledBuffer
               OpCapability ImageBuffer
               OpMemoryModel Logical GLSL450
               OpEntryPoint GLCompute %CSMain "CSMain"
               OpExecutionMode %CSMain LocalSize 1 1 1
               OpSource HLSL 630
               OpName %type_buffer_image "type.buffer.image"
               OpName %g_DispatchIndirectBuffer "g_DispatchIndirectBuffer"
               OpName %CSMain "CSMain"
               OpDecorate %g_DispatchIndirectBuffer DescriptorSet 0
               OpDecorate %g_DispatchIndirectBuffer Binding 0
       %uint = OpTypeInt 32 0
     %uint_0 = OpConstant %uint 0
     %uint_4 = OpConstant %uint 4
%type_buffer_image = OpTypeImage %uint Buffer 2 0 0 2 R32ui
%_ptr_UniformConstant_type_buffer_image = OpTypePointer UniformConstant %type_buffer_image
       %void = OpTypeVoid
          %9 = OpTypeFunction %void
%_ptr_Image_uint = OpTypePointer Image %uint
     %uint_1 = OpConstant %uint 1
%g_DispatchIndirectBuffer = OpVariable %_ptr_UniformConstant_type_buffer_image UniformConstant
     %CSMain = OpFunction %void None %9
         %12 = OpLabel
         %13 = OpImageTexelPointer %_ptr_Image_uint %g_DispatchIndirectBuffer %uint_0 %uint_0
         %14 = OpAtomicIAdd %uint %13 %uint_1 %uint_0 %uint_4
               OpReturn
               OpFunctionEnd

and then we use SPIRV-Cross to generate MSL:

#pragma clang diagnostic ignored "-Wunused-variable"

#include <metal_stdlib>
#include <simd/simd.h>
#include <metal_atomic>

using namespace metal;

kernel void CSMain(texture2d<uint> g_DispatchIndirectBuffer [[texture(0)]], device atomic_uint* g_DispatchIndirectBuffer_atomic [[buffer(0)]])
{
    uint _14 = atomic_fetch_add_explicit((device atomic_uint*)&g_DispatchIndirectBuffer_atomic[0u], 4u, memory_order_relaxed);
}

However, it creates a redundant texture2d and also changes the name of the buffer resource by appending _atomic at the end. Both of these could be avoided by doing something similar as done for RWStructuredBuffer, which would produce a regular MSL buffer of uint (not atomic_int) and would only cast to atomic where necessary (seen here). If I use SPIRV-Cross to generate HLSL instead of MSL it will produce the RWBuffer as in the original source:

RWBuffer<uint> g_DispatchIndirectBuffer : register(u0, space0);

void comp_main()
{
    uint _14;
    InterlockedAdd(g_DispatchIndirectBuffer[0u], 4u, _14);
}

[numthreads(1, 1, 1)]
void main()
{
    comp_main();
}

so it seems to have enough context to know that the texture won't be needed. This issue does not seem to be specific to DXC as I get the same output with glslang as seen here. Also to the point about context - using SPIRV-Cross reflection on the compiled MSL reports this resource as a SPVC_RESOURCE_TYPE_STORAGE_IMAGE and a dimensionality of buffer

spvc_type imageType = spvc_compiler_get_type_handle(spvCompiler, textureResource.base_type_id);        
bool isBuffer = textureDimensionality == SpvDimBuffer;  //This is true for RWBuffers and not for RWTextures

The issue seems vaguely related to https://github.com/KhronosGroup/SPIRV-Cross/issues/950.

However I don't agree with the recommendation that instead of a RWBuffer we should use a RWStructuredBuffer or RWByteAddressBuffer since they produce the desired MSL buffer. This is not an option for us at Unity as our users can write their own shaders and thus the issue can't be fixed at the HLSL level.

question rejected workaround

opened by Dredhog 18

Use constant address space for SPIR-V parameters when generating tessellation control shader.

In MoltenVK, MVKCommandEncoder::setComputeBytes can call either setBytes or setBuffer, but Metal only allows setBytes to be used with the constant address space. As a small read-only buffer, this should be a safe change for both cases.

opened by mbarriault 18

DecorationNonWritable and DecorationNonReadable are always set on Storage Images

Hi SPIRV-Cross developers:

I am a Chromium/Dawn developer and now I am meeting several issues in SPIR-V Cross when I am implementing storage textures in Dawn.

Now readonly-storage-texture and writeonly-storage-texture have been supported as WebGPU binding types, on D3D12. readonly-storage-texture is intended to match SRVs, and writeonly-storage-texture is intended to match UAVs.

But in SPIRV-Cross, I find whether I declare DecorationNonReadable or DecorationNonWritable on an image variable:

I can always detect both the decoration DecorationNonWritable and DecorationNonReadable be set. It seems here is the code in SPIRV-Cross where both these two decorations are set on all image variables.
The image variable is always translated into RWTexture2D when it is translated into HLSL.

Here is an example. Considering the following GLSL shader:

// 1.comp
#version 450
layout(set = 0, binding = 0, rgba8) uniform highp readonly image2D image0;
layout(set = 0, binding = 1, rgba8) uniform highp writeonly image2D image1;
void main() {
    vec4 pixel = imageLoad(image0, ivec2(gl_LocalInvocationID));
    imageStore(image1, ivec2(gl_LocalInvocationID), pixel);
}

Here is the translation result of 1.comp with glslang. You can see NonWritable and NonReadable has been correctly translated in the result.

// 1.spv
// glslang_validator.exe -H -V -o 1.spv 1.comp
// Module Version 10000
// Generated by (magic number): 80008
// Id's are bound by 35

                              Capability Shader
               1:             ExtInstImport  "GLSL.std.450"
                              MemoryModel Logical GLSL450
                              EntryPoint GLCompute 4  "main" 17
                              ExecutionMode 4 LocalSize 1 1 1
                              Source GLSL 450
                              Name 4  "main"
                              Name 9  "pixel"
                              Name 12  "image0"
                              Name 17  "gl_LocalInvocationID"
                              Name 27  "image1"
                              Decorate 12(image0) DescriptorSet 0
                              Decorate 12(image0) Binding 0
                              Decorate 12(image0) NonWritable
                              Decorate 17(gl_LocalInvocationID) BuiltIn LocalInvocationId
                              Decorate 27(image1) DescriptorSet 0
                              Decorate 27(image1) Binding 1
                              Decorate 27(image1) NonReadable
               2:             TypeVoid
               3:             TypeFunction 2
               6:             TypeFloat 32
               7:             TypeVector 6(float) 4
               8:             TypePointer Function 7(fvec4)
              10:             TypeImage 6(float) 2D nonsampled format:Rgba8
              11:             TypePointer UniformConstant 10
      12(image0):     11(ptr) Variable UniformConstant
              14:             TypeInt 32 0
              15:             TypeVector 14(int) 3
              16:             TypePointer Input 15(ivec3)
17(gl_LocalInvocationID):     16(ptr) Variable Input
              19:             TypeInt 32 1
              20:             TypeVector 19(int) 3
              22:             TypeVector 19(int) 2
      27(image1):     11(ptr) Variable UniformConstant
         4(main):           2 Function None 3
               5:             Label
        9(pixel):      8(ptr) Variable Function
              13:          10 Load 12(image0)
              18:   15(ivec3) Load 17(gl_LocalInvocationID)
              21:   20(ivec3) Bitcast 18
              23:     19(int) CompositeExtract 21 0
              24:     19(int) CompositeExtract 21 1
              25:   22(ivec2) CompositeConstruct 23 24
              26:    7(fvec4) ImageRead 13 25
                              Store 9(pixel) 26
              28:          10 Load 27(image1)
              29:   15(ivec3) Load 17(gl_LocalInvocationID)
              30:   20(ivec3) Bitcast 29
              31:     19(int) CompositeExtract 30 0
              32:     19(int) CompositeExtract 30 1
              33:   22(ivec2) CompositeConstruct 31 32
              34:    7(fvec4) Load 9(pixel)
                              ImageWrite 28 33 34
                              Return
                              FunctionEnd

Here is the translation result of translating 1.spv into HLSL shaders. With "--dump-resources" we can see both readonly and writeonly are set on both image0 and image1, and they are all translated into RWTexture2D.

// spirv-cross.exe 1.spv --dump-resources --hlsl --shader-model 50
...
images
=============
 ID 012 : image0 (Set : 0) (Binding : 0) writeonly readonly
 ID 027 : image1 (Set : 0) (Binding : 1) writeonly readonly
=============
...
=============
RWTexture2D<unorm float4> image0 : register(u0);
RWTexture2D<unorm float4> image1 : register(u1);

static uint3 gl_LocalInvocationID;
struct SPIRV_Cross_Input
{
    uint3 gl_LocalInvocationID : SV_GroupThreadID;
};

void comp_main()
{
    float4 pixel = image0[int2(int3(gl_LocalInvocationID).xy)];
    image1[int2(int3(gl_LocalInvocationID).xy)] = pixel;
}

[numthreads(1, 1, 1)]
void main(SPIRV_Cross_Input stage_input)
{
    gl_LocalInvocationID = stage_input.gl_LocalInvocationID;
    comp_main();
}

question

opened by Jiawei-Shao 17

MSL: Add support for writable images in iOS Tier2 argument buffers.
Add CompilerMSL::Options::argument_buffers_tier as an enumeration to allow calling app to specify platform argument buffer tier capabilities.

Support iOS writable images in Tier2 argument buffers when specified.

Tier capabilities based on recommendations from Apple engineering.
opened by billhollings 0

ivec2 offsets[4] missing from textureGatherOffsets

Converting SPIR-V corresponding to

#version 460 core
#extension GL_ARB_separate_shader_objects : enable

layout(location = 0) in vec2 inUv;

layout(location = 0) out vec4 outColor;

layout(set=0, binding=0) uniform sampler2D Image0;

void main(void)
{
    const ivec2 offs[4] = {ivec2(0,0), ivec2(1,0), ivec2(1,1), ivec2(0,1)};
    outColor = textureGatherOffsets(Image0, inUv, offs);
}

results in

#version 460

layout(binding = 0) uniform sampler2D Image0;

layout(location = 0) out vec4 outColor;
layout(location = 0) in vec2 inUv;

void main()
{
    outColor = textureGatherOffsets(Image0, inUv);
}

i.e. the offsets parameter is not filled.

In spirv_glsl.cpp (https://github.com/KhronosGroup/SPIRV-Cross/blob/master/spirv_glsl.cpp#L7271) coffsets is used for getting the correct function name, but later not for the arguments.

needs triage

opened by armatti 0

[WIP] MSL: mesh shader initial
This is more initial draft to start conversation on how-to. Mainly need to know opinion on copy-pass and SPIRVCrossDecorationInterfaceMemberIndex.

Workflow:

When cross-compiling, new types gets synthesized:

spvPerVertex all regular varyings + gl_Position

spvPerPrimitive - varyings marked as perprimitiveEXT

using spvMesh_t = mesh<spvPerVertex, spvPerPrimitive, ... >;

gl_PrimitiveTriangleIndicesEXT becomes spvMesh. Affects handling OpSetMeshOutputsEXT.

OpStore to index buffer remapped as 3 calls, in case of triangles, to spvMesh.set_index There is ugly hack to implement so - see code.

gl_MeshVerticesEXT and varyings are represented as shared memory arrays. One shader is done - they get packed into single struct. This is to work-around of SPIRV-vs-MSL api-differences. In theory, if shader writes only with [gl_LocalInvocationIndex] array can be removed.

Concerns

Had to change meaning SPIRVCrossDecorationInterfaceMemberIndex. Current implementation doesn't track array-elements - they all have same ID. On vertex/fragment seems to work fine, don't have any tesselation/geometry test-cases. I'm not sure how to go about it: current workflow with lambdas isn't suitable for

Copy pass is not really correct solution - it works only if return of entry point is called in uniform control flow. To make this work need to have something like main-wrapper, similar to HLSL backend.

Bugs/TODO:

Haven't tested gl_ClipDistance/gl_CullDistance - dunno how exactly they should work in spirv-cross. gl_PointSize is a somewhat problematic, since [[point_size]] allowed only for topology::point. Haven't test any Interpolation qualifiers.

Haven't run the test suite - sorry, it always a hassle with python scripts + Metal SDK is not present.

Performance

Apple M1 testing(in OpenGothic) shows ~2x fps regression. Can be because shared memory, but don't know for sure. Asked about this on apple forum: https://developer.apple.com/forums/thread/722047
opened by Try 4

HLSL -> SPV -> GLSL Generates invalid geometry shader (missing gl_in[])

Problem

When generating geometry shader from HLSL to GLSL in gl_PerVertex is missing gl_in[].

Reproduction steps

HLSL (test.hlsl)

#define MAX_OBJECTBUFFER_MATRIX_COUNT 64

#define SHADER_TYPE_VERTEX_SHADER 0
#define SHADER_TYPE_HULL_SHADER 1
#define SHADER_TYPE_DOMAIN_SHADER 2
#define SHADER_TYPE_GEOMRTRY_SHADER 3
#define SHADER_TYPE_PIXEL_SHADER 4
#define SHADER_TYPE_COMPUTE_SHADER 5

cbuffer _FrameBuffer : register(b0)
{
	float4x4 ProjMatrix;
	float4x4 ViewMatrix;
	
	float3 CameraPosition;
	float Time;
	float AspectRatio;
};

#if SHADER_TYPE == SHADER_TYPE_VERTEX_SHADER
cbuffer _ObjectBuffer : register(b1)
{
	float4x4 Matrices[MAX_OBJECTBUFFER_MATRIX_COUNT];
	float3 ObjectPosition;
};
#endif

struct VSInput
{
	float3 Position : POSITION0;
	float2 TextureCoordinate : TEXCOORD0;
	float3 Normal : NORMAL0;
	float4 Color : COLOR0;
	float3 BoneData : BONEINFO0;
	uint InstanceID : SV_InstanceID;
	uint VertexID : SV_VertexID;
};

struct PSInput
{
	float4 ScreenPosition : SV_POSITION;
	float3 Position : POSITION0;
	float2 TextureCoordinate : TEXCOORD0;
	float3 Normal : NORMAL0;
	float4 Color : COLOR0;
};

[maxvertexcount(6)]
void GS(point PSInput input[1], inout PointStream<PSInput> output)
{
	PSInput v = (PSInput)0;

	float2 corners[4] = {
		float2(-1.0, -1.0),
		float2(-1.0, 1.0),
		float2(1.0, 1.0),
		float2(1.0, -1.0)
	};

	float4 pos = input[0].ScreenPosition;

	// scale for depth and aspect ratio
	float depth = 0.1f / (pos.z);
	float4 scale = float4(depth, depth * AspectRatio, 0.0, 1.0);

	for (int ti = 0; ti < 2; ti++) // for each triangle
	{
		for (int vi = 0; vi < 3; vi++) // for each vertex
		{
			int ci = (ti * 2 + vi) % 3; // corner index

			v.TextureCoordinate = corners[ci];
			v.ScreenPosition = pos + float4(v.TextureCoordinate, 0.0, 1.0) * scale;
			output.Append(v);
		}
	}
	output.RestartStrip();
}

HLSL -> SPV

glslangValidator --quiet --define-macro SHADER_TYPE=3 -o test.spv -V -S geom -e GS -D test.hlsl

SPV -> GLSL

spirv-cross test.spv --version 460 --no-es --separate-shader-objects --no-support-nonzero-baseinstance --combined-samplers-inherit-bindings > test.glsl

GLSL (test.glsl)

#version 460
layout(points) in;
layout(max_vertices = 6, points) out;

in gl_PerVertex
{
    vec4 gl_Position;
}; // <-- missing gl_in[]

out gl_PerVertex
{
    vec4 gl_Position;
};

const vec2 _37[4] = vec2[](vec2(-1.0), vec2(-1.0, 1.0), vec2(1.0), vec2(1.0, -1.0));

layout(binding = 0, std140) uniform _FrameBuffer
{
    layout(row_major) mat4 ProjMatrix;
    layout(row_major) mat4 ViewMatrix;
    vec3 CameraPosition;
    float Time;
    float AspectRatio;
} _57;

layout(location = 0) out vec3 output_Position;
layout(location = 1) out vec2 output_TextureCoordinate;
layout(location = 2) out vec3 output_Normal;
layout(location = 3) out vec4 output_Color;

void main()
{
    float _178 = 0.100000001490116119384765625 / gl_in[0].gl_Position.z;
    vec4 _184 = vec4(_178, _178 * _57.AspectRatio, 0.0, 1.0);
    int _271;
    _271 = 0;
    for (; _271 < 2; _271++)
    {
        for (int _272 = 0; _272 < 3; )
        {
            int _199 = ((_271 * 2) + _272) % 3;
            gl_Position = fma(vec4(_37[_199], 0.0, 1.0), _184, gl_in[0].gl_Position);
            output_Position = vec3(0.0);
            output_TextureCoordinate = _37[_199];
            output_Normal = vec3(0.0);
            output_Color = vec4(0.0);
            EmitVertex();
            _272++;
            continue;
        }
    }
    EndPrimitive();
}

needs triage

opened by timiimit 0

SPIR-V -> HLSL : cross compiling bindings overlap

Simple reproducer:

#version 460

layout (location = 0) out vec4 fragColor;

layout(binding = 0, std430) readonly buffer A { vec4 a[]; };
layout(binding = 0, std430) readonly buffer B { uint b[]; };

void main()
{
	fragColor = vec4(0.4, 0.4, 0.8, 1.0);
}

Been cross-compiled to HLSL as:

// error: resource _22 at register 0 overlaps with resource _17 at register 0, space 0
ByteAddressBuffer _17 : register(t0);
ByteAddressBuffer _22 : register(t0);
...

Seemingly there is 'simple' solution for SSBO - to use only a single binding. However, I don't know, if there is a solution for textures. Any opinions on should this be implemented in any way or not? Thanks!

opened by Try 1