nicegraf-shaderc is a command-line tool that transforms HLSL code into shaders for various graphics APIs.

Related tags

CLI niceshade
Overview

Build status Build Status

User Manual

Table of Contents

Introduction

nicegraf-shaderc is a command-line tool that transforms HLSL code into shaders for various graphics APIs. Presently, the following APIs can be targeted:

  • OpenGL 4.3
  • OpenGL ES 3.1+
  • Metal 1.0+
  • Vulkan 1.0+

The input HLSL files may contain definitions of several entry points for different shader stages. The entry points can be configured into a single rendering pipeline (with additional options, if desired) using a special directive. For each of these configurations (called techniques), the tool will generate platform-specific shaders.

As an example, here is a shader that calculates the relative luminance of each pixel in an image. For demonstration purposes, it allows to optionally apply gamma correction to input and/or output.

// The comments below are recognized by nicegraf-shaderc as technique definitions.
//T: relative-luminance vs:VSMain ps:PSMain define:OUTPUT_NEEDS_GAMMA_CORRECTION=1 define:INPUT_NEEDS_GAMMA_CORRECTION=1
//T: relative-luminance-srgb-texture vs:VSMain ps:PSMain define:OUTPUT_NEEDS_GAMMA_CORRECTION=1
//T: relative-luminance-srgb-framebuffer vs:VSMain ps:PSMain define:INPUT_NEEDS_GAMMA_CORRECTION=1
//T: relative-luminance-srgb-texture-and-framebuffer vs:VSMain ps:PSMain

float4 VSMain(uint vid : SV_VertexID) : SV_POSITION { // vertex shader
  const float2 fullscreen_triangle_verts[] = {
    float2(-1.0, -1.0), float2(3.0, -1.0), float2(-1.0,  3.0)
  };
  return  float4(fullscreen_triangle_verts[vid % 3], 0.0, 1.0);
}

uniform Texture2D img;

float4 PSMain(float4 frag_coord : SV_POSITION) : SV_TARGET { // pixel shader
  const float gamma = 2.2;
  uint img_width, img_height;
  img.GetDimensions(img_width, img_height);
  float3 color = img.Load(int3(int2(frag_coord.xy) % int2(img_width, img_height), 0)).rgb;
#if defined(INPUT_NEEDS_GAMMA_CORRECTION)
  color = pow(color, gamma);
#endif
  float relative_luminance = dot(float3(0.2126, 0.7152, 0.0722), color);
#if defined(OUTPUT_NEEDS_GAMMA_CORRECTION)
  relative_luminance = pow(relative_luminance, 1.0 / gamma);
#endif
  return float4(float3(relative_luminance, relative_luminance, relative_luminance), 1.0);
}

In addition to generating shaders, the tool captures and writes out the information about resources (textures, buffers, etc.) used by each technique defined in the input file. This information can be used by the application for various purposes, such as streamlining Vulkan pipeline layout creation.

This tool is powered by Microsoft DirectXShaderCompiler and SPIRV-Cross.

Project Status

This project is under active development. Some features may change significantly.

Planned Features

  • Adding support for hull, domain, geometry and compute shaders;
  • Batching and parallel compilation to speed up builds.

Obtaining the Source Code and Building

You will need to have git and cmake installed on your system. On Windows, building with compilers other than MSVC may not work.

Execute the following command to clone the project's repository:

git clone https://github.com/nicebyte/nicegraf-shaderc.git

Once the cloning process is complete, execute the following commands from the root of the repository:

mkdir build
cd build
cmake .. -Ax64

This will generate project files specific to your system in the build folder. After building the generated project, the nicegraf_shaderc binary can be found in the repository's root folder.

Running

To transform an input HLSL file to platform-specific shaders, execute:

nicegraf_shaderc

Valid command line options are:

  • -O - specifies the folder to store the output files in. By default, the output files are written to the current working directory.
  • -t - specifies a target to generate shaders for. Accepted values are:
    • gl430 for OpenGL;
    • gles310, gles320 for OpenGL ES;
    • msl10, msl11, msl12, msl20 for Metal on macOS;
    • msl10ios, msl11ios, msl12ios, msl20ios for Metal on iOS;
    • spv for SPIR-V.
  • -o - Set SPIR-V optimization level. 1 will apply the same optimizations as spirv-opt -O. 0 will turn off all optimizations. The default value is 0. SPIR-V optimizations have an effect on the output for non-SPIR-V targets. Enabling them will result in less readable output that doesn't match the input HLSL as closely. Using code generated from optimized SPIR-V may result in performance wins on some platforms, but, as always, measure.
  • -m - HLSL shader model version to use. Valid values are: 6_0, 6_1, 6_2, 6_3, 6_4, 6_5, 6_6. Default is 6_0.
  • -h - Path (relative to the output folder) for the generated header file with descriptor binding and set numbers. If not specified, no header file will be generated.
  • -n - Namespace for the generated shader file. If not specified, global namespace is used.
  • -D = - Add a preprocessor definition name with the value value to techniques.

Shaders will be generated for each of the techniques specified in the input file and each of the targets specified in the command line options.

For example, the following line will produce OpenGL 4.3 and Metal 1.2 shaders for each technique defined in input.hlsl, in the generated_shaders/ subfolder:

nicegraf_shaderc input.hlsl -O generated_shaders/ -t gl430 -t msl12

Defining Techniques

Techniques are defined using a special comment:

//T:

The technique name may include alphanumeric characters, underscores (_) and dashes (-).

A tag is a name-value pair separated by a colon. For example, vs:VSMain is a tag, vs is the tag name and VSMain is the value.

The following tag names are valid:

  • vs - the tag value specifies the entry point for the vertex shader stage;
  • ps - the tag value specifies the entry point for the pixel shader stage;
  • define - the tag value specifies an additional preprocessor definition;
  • meta - the tag value specifies an additional metadata entry. It should be a name-value pair separated by a = sign, i.e.: meta:enable_depth_testing=1. These values get stored as part of the pipeline metadata file (see below) and users are free to interpret them as they wish.

A valid technique definition must at least specify an entry point for the vertex stage.

Generated Header File

Using the -h command line option, you may specify a special C++ header file to be generated as part of the shader compilation process. The said file shall contain named constants for all descriptor bindings and sets used by different techniques defined in the input. The constants for each technique are put into their own namespace, named after the technique (hyphens in tenchnique names are replaced by underscores to get valid C++ identifiers). Additionally, you may put the entire contents of the generated header into another namespace, specified by the -n command line option.

Below is an example of an input file and the generated header it produces.

Input file:

//T: imgui ps:PSMain vs:VSMain

struct ImGuiVSInput {
  float2 position : ATTR0;
  float2 uv : TEXCOORD0;
  float4 color : COLOR0;
};

struct ImGuiPSInput {
  float4 position : SV_POSITION;
  float2 uv : TEXCOORD0;
  float4 color : COLOR0;
};

cbuffer MatUniformBuffer : register(b0){
  float4x4 u_Projection;
}

[[vk::binding(1, 0)]] uniform Texture2D u_Texture;
[[vk::binding(2, 0)]] uniform sampler u_Sampler;

ImGuiPSInput VSMain(ImGuiVSInput input) {
  ImGuiPSInput output = {
    mul(u_Projection, float4(input.position, 0.0, 1.0)),
    input.uv,
    input.color
  };
  return output;
}

float4 PSMain(ImGuiPSInput input) : SV_TARGET {
  return input.color * u_Texture.Sample(u_Sampler, input.uv);
}

Generated header:

/*auto-generated, do not edit*/
#pragma once
namespace shader_consts {
namespace imgui {
  static constexpr int u_Sampler_Binding = 2;
  static constexpr int u_Sampler_Set = 0;
  static constexpr int u_Texture_Binding = 1;
  static constexpr int u_Texture_Set = 0;
  static constexpr int MatUniformBuffer_Binding = 0;
  static constexpr int MatUniformBuffer_Set = 0;
}
}

Pipeline Metadata

For each technique defined in the input file, nicegraf_shaderc will produce a corresponding .pipeline file, which contains the following information:

  • List of all resources consumed by the technique, including their types, bindings and which pipeline stages they are used by;
  • A mapping from separate image and sampler bindings to auto-generated combined image/sampler bindings (relevant for targets which don't have full separation between textures and samplers at the shader level, i.e. OpenGL);
  • Any additional metadata provided by the user in the technique specification using the meta: tag.

.pipeline files are binary. Code for parsing the binary format is provided in the metadata_parser subfolder of the source code repository. Alternatively, .pipeline files can be converted to human-readable JSON using the display_metadata utility, source code for which is provided in the samples subfolder of the repository. A detailed description of the metadata file format is provided below.

Using Vulkan features from HLSL

You may choose to use the Vulkan binding model explicitly and assign descriptor sets and bindings like this:

[[vk::binding(1, 0)]] uniform Texture2D tex; // assign tex to set 0 binding 1
[[vk::binding(2, 0)]] uniform sampler samp;  // assign tex to set 0 binding 2

You may use specialization constants as well:

[[vk::constant_id(1)]] const float specConstFloat = 1.5;

See here for more details.

Pipeline Metadata File Format

For each technique described in the input file, nicegraf-shaderc emits a file containing information that can be leveraged to simplify pipeline creation. This data includes:

  • Description of the pipeline layout;
  • Mapping from separate image and sampler bindings to their corresponding auto-generated combined image/sampler bindings (for platforms that don't have full separation between textures and samplers, i.e. OpenGL);
  • Any additional metadata specified by the user using the meta: tag in the technique description.

A detailed description of the file's format follows.

General Conventions

A pipeline metadata file (referred to simply as "file" henceforth) is broken into records. A record is a sequence of fields and raw byte blocks. A field is a 4-byte unsigned integer in network byte order. A raw byte block consists of a header and a body. The body of a raw byte block is an arbitrary sequence of bytes. The length of the body is always a multiple of 4 bytes. The header of a raw byte block is two 32-bit unsigned integers in network byte order. The first is always equal to 0xffffffff and signifies the beginning of a raw byte block. The second specifies the length of the block's body divided by 4.

A record's type is defined by its layout. The following types of records are defined:

  • HEADER;
  • ENTRYPOINTS;
  • PIPELINE_LAYOUT;
  • SEPARATE_TO_COMBINED_MAP;
  • USER_METADATA.

A detailed description of each record type follows.

The HEADER Record Type

The very first record in a file is always of the HEADER type. There is always only one instance of a HEADER record in a file.

A HEADER record contains the following fields, in this exact order:

  • magic_number - valid files must always have this field set to 0xdeadbeef;
  • header_size - total size of the header record in bytes (including the header_size and magic_number fields);
  • version_maj - major version number of the metadata format in use;
  • version_min - minor version number of the metadata format in use;
  • entrypoints_offset - offset, in bytes, from the beginning of the file, at which the ENTRYPOINTS record is stored;
  • pipeline_layout_offset - offset, in bytes, from the beginning of the file, at which the PIPELINE_LAYOUT record is stored;
  • image_to_cis_map_offset - offset, in bytes, from the beginning of the file, at which a SEPARATE_TO_COMBINED_MAP record is stored, which maps separate image bindings to the corresponding auto-generated combined image/sampler bindings;
  • sampler_to_cis_map_offset - offset, in bytes, from the beginning of the file, at which a SEPARATE_TO_COMBINED_MAP record is stored, which maps separate sampler bindings to the corresponding auto-generated combined image/sampler bindings;
  • user_metadata_offset - offset, in bytes, from the beginning of the file, at which the USER_METADATA record is stored;

The ENTRYPOINTS Record Type

This record type provides the names of entry points for individual shader stages involved in the technique.

The first field in this record, num_entrypoints, contains the number of entrypoints, one per shader stage. It is followed by a sequence of num_entrypoints entrypoint descriptions.

An entrypoint description consists of a field, type which indicates the type of the shader stage the entrypoint is for (0 for vertex shader, 1 for fragment shader), and a raw byte block containing a null-terminated entrypoint name.

The PIPELINE_LAYOUT Record Type

This record type provides information about descriptor sets, descriptor types and bindings used by the pipeline. There is always only one instance of a PIPELINE_LAYOUT record in a file.

A PIPELINE_LAYOUT record contains the following fields, in this exact order:

  • num_descriptor_sets - total number of descriptor sets in the layout;
  • For each of the num_descriptor_sets descriptor sets:
    • num_descriptors - number of descriptors in the set;
    • For each of the num_descriptors descriptors:
      • binding_id - bindning number of the descriptor;
      • descriptor_type - an integer indicating the type of the descriptor. The following values are valid:
        • 0x00 - indicates a uniform buffer;
        • 0x01 - indicates a storage buffer;
        • 0x02 - indicates a load/store image;
        • 0x03 - indicates a texture;
        • 0x04 - indicates a sampler;
        • 0x05 - indicates a combined texture/sampler.
      • 0x06 - indicates a uniform texel buffer.

The SEPARATE_TO_COMBINED_MAP Record Type

Some platforms do not have full separation between textures and samplers. For example, in OpenGL, in order to sample a texture with a particular sampler, both the texture and the sampler need to be bound to the same texture unit by the CPU-side code. This is in contrast to HLSL, which allows specifying the sampler to use directly from the shader code.

To address this discrepancy, each unique texture-sampler pair used by the source HLSL generates a "synthetic" combined texture/sampler in the output. Each separate texture and sampler is then mapped to a set of auto-generated combined texture/samplers that it is used in.

A SEPARATE_TO_COMBINED_MAP record contains the following fields, in this exact order:

  • num_entries - number of entries in the record;
  • For each entry:
    • set_id - descriptor set id of the separate image or sampler;
    • binding_id - binding id of the separate image or sampler;
    • num_combineds - number of combined texture/samplers that the separate image or sampler is being used by;
    • For each of the num_combineds combined texture/samplers:
      • combined_binding_id - binding id of the auto-generated combined image/sampler;

The USER_METADATA Record Type

This record stores any additional user metadata specified by meta: tags in the technique description.

The USER_METADATA record has only one field, num_metas. Following the field are num_metas pairs of raw byte blocks. The first block in a pair stores the user-provided key, and the second stores the user-provided value (both are null-terminated strings).

Owner
nicebyte
profile pic is my face when debugging code from last night in the morning
nicebyte
led is a line-oriented text editor in command line

led is a line-oriented text editor in command line. This editor is similar to the standard program on unix systems - GNU ed. But i'm not going to make an exact clone of that program, it's just a pet project.

Artem Mironov 9 Jun 14, 2022
A command-line tool to generate Linux manual pages from C source code.

mangen A command-line tool to generate Linux manual pages from C source code. Description mangen is, as said above, a program to generate Linux manual

null 2 Nov 15, 2021
pbr2gltf2 is a command line tool for converting PBR images to a glTF 2.0 material.

pbr2gltf2 is a command line tool for converting PBR images to a glTF 2.0 material. The tool is detecting depending on the filename, which PBR information is stored. It swizzles the images and does reassign the channels to a glTF 2.0 image. The tool stores the images plus a minimal, valid glTF 2.0 file containing the required material, textures and images.

UX3D GmbH 22 Jan 11, 2022
A command-line tool to display colorful distro information.

sjfetch A command-line tool to display colorful distro information.

Fikret Musk 6 Apr 6, 2022
Simple command line tool that processes image files using the FidelityFX Super Resolution (FSR) or Contrast Adaptive Sharpening (CAS) shader systems.

Simple command line tool that processes image files using the FidelityFX Super Resolution (FSR) or Contrast Adaptive Sharpening (CAS) shader systems.

GPUOpen Effects 174 Jun 26, 2022
A command line tool with no external dependencies to print information about an X server instance.

xinfo A command line tool with no external dependencies to print information about an X server instance. Building and running To build the code in thi

Jean-Michel Gorius 6 Jan 13, 2022
A command line tool for numerically computing Out-of-time-ordered correlations for N=4 supersymmetric Yang-Mills theory and Beta deformed N=4 SYM.

A command line tool to compute OTOC for N=4 supersymmetric Yang–Mills theory This is a command line tool to numerically compute Out-of-time-ordered co

Gaoli Chen 1 Oct 16, 2021
A command-line tool to extract dylib files from the dyld shared cache file.

DyldExtractor A command-line tool to extract dylib files from the dyld shared cache file. Starting with macOS 11, standalone binaries of system librar

Cyandev 8 Mar 12, 2022
Microsoft Visual TrueType(VTT) command line compile tool.

Project Microsoft Visual TrueType(VTT) is a professional-level tool for graphically instructing TrueType and OpenType fonts. For details on the tool v

Microsoft 34 May 19, 2022
brn is a command line tool similar to vimv.

brn is a command line tool similar to vimv. It can be used to easily mass-rename files in your preferred text editor (i.e. vim).

Nimai Patel 12 Feb 26, 2022
A single header C++ library for parsing command line arguments and options with minimal amount of code

Quick Arg Parser Tired of unwieldy tools like getopt or argp? Quick Arg Parser is a single header C++ library for parsing command line arguments

null 44 Feb 21, 2022
null 76 Apr 18, 2022
A simple command line application in order to create new Code workspaces.

mkcws Summary A simple command line application in order to create new Code workspaces. License This project's license is GPL 2. The whole license tex

Kevin Matthes 0 Apr 1, 2022
A simple to use, composable, command line parser for C++ 11 and beyond

Clara v1.1.5 !! This repository is unmaintained. Go here for a fork that is somewhat maintained. !! A simple to use, composable, command line parser f

Catch Org 651 Jun 15, 2022
A library for interactive command line interfaces in modern C++

cli A cross-platform header only C++14 library for interactive command line interfaces (Cisco style) Features Header only Cross-platform (linux and wi

Daniele Pallastrelli 803 Jul 2, 2022
CLI11 is a command line parser for C++11 and beyond that provides a rich feature set with a simple and intuitive interface.

CLI11: Command line parser for C++11 What's new • Documentation • API Reference CLI11 is a command line parser for C++11 and beyond that provides a ri

null 2.1k Jun 24, 2022
Lightweight C++ command line option parser

Release versions Note that master is generally a work in progress, and you probably want to use a tagged release version. Version 3 breaking changes I

null 3.1k Jun 25, 2022
A simple to use, composable, command line parser for C++ 11 and beyond

Lyra A simple to use, composing, header only, command line arguments parser for C++ 11 and beyond. Obtain License Standards Stats Tests License Distri

Build Frameworks Group 342 Jun 16, 2022
CLIp is a clipboard emulator for a command line interface written in 100% standard C only. Pipe to it to copy, pipe from it to paste.

CLIp v2 About CLIp is a powerful yet easy to use and minimal clipboard manager for a command line environment, with no dependencies or bloat. Usage Sy

A.P. Jo. 12 Sep 18, 2021