LLVM IR and optimizer for shaders, including front-end adapters for GLSL and SPIR-V and back-end adapter for GLSL

Overview

Licensing

LunarGLASS is available via a three clause BSD-style open source license.

Goals

The primary goals of the LunarGLASS project are:

  • Reduce the developement burden of creating advanced shader compiler stacks.
  • Increase the level of optimization achieved by real world applications. and do all this in a robust and modular approach that works well across a diverse set of hardware architectures.

Description

LunarGLASS is an LLVM-based shader-compiler stack available to open-source developers. It brings a new approach by splitting the common shared intermediate representation (IR) into two levels; the top level is completely platform independent while the bottom level is dynamically tailorable to different families of architecture. Both levels still lend themselves to portability and sharing of tools. Together, they solve the problem of having a standard portable IR without being biased toward a specific class of target architecture.

LunarGLASS is a long-term compiler stack architecture, based on establishing common intermediate representations (IRs) allowing modularity between stack layers. Each source-language front end would benefit from a common set of high- and mid-level optimizations, as would each back end, without the need to invent additional IRs. The short-term goal is to leverage investments in existing IRs while the long-term goal is to reduce the number of IRs and not require optimization difficulties caused by losing information going through an IR.

Downloading and Building

The standard way to build is using glslang as the GLSL front end. This is currently done by making them sibling directories:

PathOfYourChoice/glslang
PathOfYourChoice/LunarGLASS

(See https://github.com/KhronosGroup/glslang for details about glslang.)

Getting the code

  1. In PathOfYourChoice, clone the glslang repository from https://github.com/KhronosGroup/glslang, making the glslang subdirectory.
  2. In PathOfYourChoice, clone the LunarGLASS repository from https://github.com/LunarG/LunarGLASS, making the LunarGLASS subdirectory.
  3. Get LLVM. Download the LLVM 3.4 source code from http://llvm.org/releases/download.html#3.4 into your "LunarGLASS/Core/LLVM" directory, then extract it:
cd LunarGLASS/Core/LLVM
tar --gzip -xf llvm-3.4.src.tar.gz
  1. The previous step overrwrote some LLVM files that LunarGLASS changes. Restore them to LunarGLASS's versions, while still in the LunarGLASS/Core/LLVM directory:
git checkout -f .  # put back the LunarGLASS versions of some LLVM files

Building

LunarGLASS must be built after glslang and LLVM have been built.

Building glslang

Use CMake, building in the directory PathOfYourChoice/glslang/build:

cmake -DCMAKE_INSTALL_PREFIX="" ..
make
make install DESTDIR=install

Building LLVM

Building LLVM for Windows

  1. Use version 2.7.6 of python (Get it from http://www.python.org/download/releases/2.7.6/.)
  2. Run CMake.
  3. Put the full path to your LunarGLASS/Core/LLVM/llvm-3.4 location in "Where is the source code:" and add "/build" to it (LunarGLASS/Core/LLVM/llvm-3.4/build) in "Where to build the binaries:"
  4. Press "Configure" button in CMake and say yes to create the build directory.
  5. Select your Visual Studio and "Use default native compilers" and "Finish".
  6. Change CMAKE_INSTALL_PREFIX to "install" (no path).
  7. Press "Configure" again.
  8. Press "Generate" button in CMake.
  9. Open the LLVM.sln just created in llvm-3.4/build and build the INSTALL project.

Building LLVM for Linux

Summary: LLVM uses a configure script while glslang and LunarGLASS use CMake.

  1. First build set up:
# first time only
cd LunarGLASS/Core/LLVM/llvm-3.4
mkdir build
cd build
../configure
  1. Build:
# build or rebuild
cd LunarGLASS/Core/LLVM/llvm-3.4/build
make -j 8
make install DESTDIR=`pwd`/install

Building LunarGLASS (the LunarGOO standalone tool)

By default, this builds a command-line tool that translates GLSL -> LunarGLASS -> GLSL.

Use CMake, building in the directory PathOfYourChoice/LunarGLASS/build.

Modes of use

Use a LunarGLASS stack to compile from a driver:

Per driver initialization:

  1. Include LunarGLASSManager.h. This header includes no other headers.
  2. Use gla::getManager() to get a manager. (From below, your compiler implementation will have derived a concrete class and factory for it.)

Per compile:

  1. Pass this manager to a front end that builds LunarGLASS Top IR and saves the llvm module into the manager using manager->setModule() and saves symbol tables using manager->set*Symbols().
  2. Use manager->translateTopToBottom() to create bottom IR.
  3. Use manager->translateBottomToTarget() to translate that to the compiler's back end target language.
  4. Use manager->clear() to free up structures specific to the compile, e.g., things set in step 3.

Make a LunarGLASS back end:

  1. Make a manager that derives from the gla::PrivateManager in PrivateManager.h and provides a factory gla::getManager() that makes one of these.
  2. Make a back end that derives from the gla::BackEnd() in Backend.h and have it supply methods to describe what form of Bottom IR it wants to consume.
  3. Also have your back end derive from gla::BackEndTranslator() to fill in all the methods needed to translate from Bottom IR to your target.
  4. You will need to include BottomIR.h to consume the details of Bottom IR. Between this file, the LunarGLASSTopIR.h file it includes, and the specification, you have everything you need to interpret Bottom IR.
  5. Your private manager (from 1) will have to make one of these back ends (from 2) and back end translators (from 3) when it is created.

Make a LunarGLASS front end:

  1. Include LunarGLASSManager.h and LunarGLASSTopIR.h in your front end. Between these and the specification, you have everything you need to create Top IR.
  2. Call your front end (as in step 3 in "Use a LunarGLASS stack...") to translate your source language to Top IR and leave the llvm module in the manager.
Comments
  • Hexagon - Broken Module

    Hexagon - Broken Module

    During the optimization phase (TopToBottom.cpp : runLLVMOptimizations1) some Hexagon intrinsics are generated, which result in an assert:

    Incorrect number of arguments passed to called function! %1 = call i32 @llvm.hexagon.A2.tfrih.v4f32.v4f32.v3f32.v3f32.v3f32.f32(<4 x float> undef, i32 15, <3 x float> %0, i32 0, <3 x float> %0, i32 1, <3 x float> %0, i32 2, float 1.000000e+00, i32 0) Stored value type does not match pointer operand type! store i32 %1, <4 x float>* @_inreturn0, align 16 <4 x float>Broken module found, compilation aborted!

    The store i32 %1, <4 x float>* @_inreturn0, align 16 section is not my doing, I assume it's something the optimizer changed from this line in the dumped IR: store <4 x float> %14, <4 x float>* @_inreturn0

    The error is triggered from this section (dumped ir): %7 = load <3 x float>* %geptr %ptr = extractelement <3 x float> %7, i32 0 %8 = insertelement <4 x float> undef, float %ptr, i32 0 %ptr4 = extractelement <3 x float> %7, i32 1 %9 = insertelement <4 x float> %8, float %ptr4, i32 1 %ptr5 = extractelement <3 x float> %7, i32 2 %10 = insertelement <4 x float> %9, float %ptr5, i32 2 %11 = insertelement <4 x float> %10, float 1.000000e+00, i32 3 store <4 x float> %11, <4 x float>* %geptr2

    However I see nothing directly illegal by doing the above, then again I'm new to the LLVM scene.

    opened by miguel-petersen 7
  • glslang master ENABLE_OPT CMakeLists.txt option needed

    glslang master ENABLE_OPT CMakeLists.txt option needed

    The newest version of glslang master adds ENABLE_OPT which defaults to ON. LunarGLASS CMakeLists.txt needs to be updated with this option and include SPIV-Tools, SPIRV-Tools-opt, and SPVRemapper when it's enabled. Right now it gets a link error.

    opened by danginsburg 4
  • LunarGOO generates invalid GLSL for arrays in shader storage buffers

    LunarGOO generates invalid GLSL for arrays in shader storage buffers

    Hi, I have tried to compile some of my shaders with LunarGOO to generate optimized shaders. However, the glsl generator seems to have problems with arrays in shader storage buffers. I have attached a test geometry shader, which generate the following faulty output:

    #version 430 core
    // LunarGOO output
    #extension GL_ARB_enhanced_layouts : enable
    layout(points) in;
    layout(triangle_strip) out;
    layout(max_vertices = 3) out;
    layout(std430) buffer VertexColours {
      vec4 Colour[];
    } ;
    layout(location=0) in VertexData {
      ivec3 Indices;
    } In[1];
    out VertexData {
      vec4 Colours;
    } Out;
    
    void main()
    {
      Out.Colours = [In[0].Indices.x].Colour;
      EmitVertex();
      Out.Colours = [In[0].Indices.y].Colour;
      EmitVertex();
      Out.Colours = [In[0].Indices.z].Colour;
      EmitVertex();
      EndPrimitive();
    }
    

    buffers_bug.geom.txt

    opened by bfierz 3
  • Fix compile errors and assert when using latest glslang:

    Fix compile errors and assert when using latest glslang:

    • Compile fixes for TType::isRuntimeSizedArray() being removed and computeTypeLocationSize() requiring the stage.
    • TType::getStruct() was changed to assert(isStruct()) which meant several places causing getStruct() would assert in debug builds. Add getStructIfIsStruct() and use that or isStruct() in places where checking for nullptr was used.
    opened by danginsburg 2
  • Current effort?

    Current effort?

    Is there a current github repo of code that has this capability? I wrote C++->SPIRV and am looking for SPIRV->SPIRV optimization. Hopefully something I can link against LLVM-8 which I'm already including in my binary.

    opened by seanbaxter 2
  • Why my program always tell me “Segmentation fault” ????

    Why my program always tell me “Segmentation fault” ????

    I am very sorry to bother you again and again. I used the command in terminal: ~/Downloads/test/LunarGLASS/build/install/bin$ ./LunarGOO 100ops.frag -t Then the program tell me "Segmentation fault(core dump)" The file 100ops.frag is in your LunarGLASS/test/ folder I don't know if this error is due to a problem with my file or because of a problem with my operation.

    100ops.frag : **` #version 100

    lowp float foo();

    uniform int low, high;

    lowp float face1 = 11.0;

    void main() { int z = 3;

    if (2 * low + 1 < high)
        ++z;
    
    gl_FragColor = face1 * vec4(z) + foo();
    

    }

    lowp float face2 = -2.0;

    lowp float foo() { // testing if face2 initializer insert logic is correct in main return face2; }`**

    opened by SakuraXiaMF 2
  • glslang pointed by the documentation is missing some libs needed by LunarGLASS

    glslang pointed by the documentation is missing some libs needed by LunarGLASS

    SPIRV-Tools.lib you need for LunarGOO project is missing from https://github.com/KhronosGroup/glslang : the one you ask to use in your readme

    On the other hand, SPIRV-Tools libs are available from glslang shipped with Vulkan 1.1.97.0... But if I try to use it instead, I get other errors during the build.

    Surprsing that https://github.com/KhronosGroup/glslang doesn't seem to provide the same as the glslang shipped with VulkanSDK 1.1.97.0

    opened by tlorach 2
  • LunarGLASS does not compile with latest glslang master

    LunarGLASS does not compile with latest glslang master

    The following commit in glslang master removed functions that LunarGOO is using:

    https://github.com/KhronosGroup/glslang/commit/a558b26537e47068f027856ab60a9c3ba45ffdbe#diff-9f27b0011955ee30340cd6e37aa3b7b4

    opened by danginsburg 2
  • Add ENABLE_NV_EXTENSIONS and ENABLE_HLSL to CMakeLists.txt

    Add ENABLE_NV_EXTENSIONS and ENABLE_HLSL to CMakeLists.txt

    LunarGLASS has gotten out of sync with latest glslang master. This causes crashes in LunarGOO due to mismatched #define's compiling the two. The fix is to add the following to CMakeLists.txt in LunarGLASS:

    option(ENABLE_NV_EXTENSIONS "Enables support of NV-specific extensions" ON) option(ENABLE_HLSL "Enables HLSL input support" ON) if(ENABLE_NV_EXTENSIONS) add_definitions(-DNV_EXTENSIONS) endif(ENABLE_NV_EXTENSIONS)

    if(ENABLE_HLSL) add_definitions(-DENABLE_HLSL) endif(ENABLE_HLSL)

    opened by danginsburg 2
  • LunarGOO producing incorrect result for shader and Unsupported functionality

    LunarGOO producing incorrect result for shader and Unsupported functionality

    The following shader:

    #version 140
    
    uniform int g_nDebugMode;
    uniform int g_nTest;
    uniform float g_flTest;
    
    uniform vec4 result0;
    uniform vec4 result1;
    uniform vec4 result2;
    uniform vec4 result3;
    vec4 ApplyDebugMode( out bool bReturn )
    {
        vec4 vResult;
        if ( g_nDebugMode == 0 )
        {
            bReturn = true;
            vResult = result0;
        }
        else
        {
            bReturn = false;
            vResult = result1;
        }
        return vResult;
    }
    
    vec4 foo()
    {
        vec4 vOutColor;
        if ( g_nDebugMode != 1 )
        {
            bool bReturn = true;
            vOutColor = ApplyDebugMode( bReturn );
            if ( bReturn )
                return vOutColor;
        }
    
        if ( g_flTest > 0.0 )
        {
            if ( g_nTest > 0 )
            {
                vOutColor = result2;
            }
            else
            {
                vOutColor = result3;
            }
    
        }
        return vOutColor;
    }
    
    void main()
    {
        gl_FragColor = foo();
    }
    

    Produces the following output in LunarGOO:

    ***Unsupported functionality: general-case cross edges (requires artificial loop )

    And then also produces incorrect output code if the warning is ignored.

    opened by danginsburg 2
  • Moving to a full location/binding model in metadata and GLSL output

    Moving to a full location/binding model in metadata and GLSL output

    For historical reasons, the GLSL back end intentionally suppresses issuing of layout(binding=...), and binding numbers shared the metadata with location numbers, as if they were mutually exclusive. This does not scale into the future. LunarGOO should correctly output layout(binding=...), and when both location and binding information is needed, two values are needed in the metadata, not one.

    Proposed fix: Widen the metadata (!typeLayout) to add on binding, as well as memory qualifiers (writeonly/restrict/etc.). See Core/metadata.h. When this widened data is present, binding would no longer reside in the location operand, but in the binding operand. Additionally, always have the GLSL back end correctly emit binding numbers.

    This makes a whole new class of shaders work correctly through the stack. Will look at providing mechanisms to enable the legacy behavior, but best will be to switch to the correct representation and output.

    Summary of visible effects

    • metadata getting wider, generally in a backward compatible way, except for binding values
    • GLSL output will have correct location and binding values
    • GLSL output will correctly include #extension GL_ARB_shading_language_420pack, which is required to use layout(location=) in the shader
    opened by johnkslang 2
  • How to generate GPU backend binaries

    How to generate GPU backend binaries

    Hello, my friends, @svenstaro @danginsburg @johnkslang @cnorthrop @jeremy-lunarg I'm a graphics beginner, learning how to translate OpenGL SL to a target(like NV/MALI/AMD-GPU) instructions binnary, I see your tools and successfully install it on my Linux, with minor changes.

    change 1

    add .a files to PathOfYourChoice/LunarGLASS$/CMakeLists.txt
    add all .a files to the libs under glslang/build/install/lib
    set(GLSLANGLIBS
    	GenericCodeGen
    	glslang
    	glslang-default-resource-limits
    	HLSL
    	MachineIndependent
    	OGLCompiler
    	OSDependent
    	SPIRV)
    

    change 2

    //remove the 2 un-necessary lines in  PathOfYourChoice/LunarGLASS/Standalone/main.cpp
    //#include "glslang/Include/revision.h
    //printf("Glslang Version: %s %s\n", GLSLANG_REVISION, GLSLANG_DATE);
    

    changes 3

    SET SPRIV .a files LIBRARY_PATH
    export LIBRARY_PATH=/home/myworkdir/3RD_PART/SPIRV-Tools/build/install/usr/local/lib
    

    install and run success

    ./LunarGOO -a test.vert
    

    with above changes and command , I successfully generate Top-IR and Bottom-IR for my test.vert shader. and I see the LunarGLASS/Core/LLVM/llvm-3.4/lib/Target/R600 is ADM-R600-GPU backend.

    Questions

    How can I translate test.frag or test.vert to ADM-R600-GPU binnaris? would you give me guidance. Q1. How to use LLVM inline all functions? Q2. Where should I add AMD-R600 backend pass?

    opened by xiaoqiang-wang 1
  • Some problems about this program

    Some problems about this program

    I very glad to you share your program.This program is very important for my job. I have some problems about this program.

    Initially,In your program brief,this program can translate GLSL code to LLVM IR,but you have Top IR and Bottom IR,who is LLVM IR ? Next,I read your Readme.td file,but I don't know how to translate GLSL file to LLVM .bc file?Your program have this function?

    opened by xiaxiaxiahhh 1
  • How to translate GLSL file to LLVM's .bc file?

    How to translate GLSL file to LLVM's .bc file?

    Hey,I am asking question again~~

    My LLVM version is LLVM-4.0 Initially,I saved 130.frag.out file's bottom IR part as a.ll file. Then, I use llvm-as to translate a.ll file to a.bc file, but I get an error :"error: expected comma after load's type %0 = load i32 addrspace(1)* @sampC, !gla.uniform !50"

    Then I saved 130.frag.out file's Top IR part as b.ll file.Next,I use llvm-as to translate b.ll file to b.bc file,I get an error again: "error: expected comma after getelementptr's type %1 = call i32 @cudaSetupArgument(i8* %0, i64 ptrtoint (i32* getelementptr (i32* null, i32 1) to i64), i64 0)"

    So,our program how to translate GLSL file to LLVM's .bc file??

    opened by SakuraXiaMF 4
  • How to use this program??

    How to use this program??

    Sorry, I don't understand how to use the LunarGLASS. When I reading the readme.td, which is looks like I can't use terminal command to translate a .vert file .geom file or .frag file to LLVM IR.I need to write a demo that use LuanrGLASS's function to translate .vert file to LLVM IR.

    opened by SakuraXiaMF 0
  • Appreciate for you to shared your program.

    Appreciate for you to shared your program.

    I am so sorry to bother you a lots of times in issues.I try a lots of times to build LunarGLASS,but I failed a lots of times.The most error is something in namespace glslang doesn't name a type or something is not a member of glslang. For example: LunarGLASS/Frontends/glslang/GlslangToTopVisitor.cpp:105:86: error: ‘TCrackedTextureOp’ in namespace ‘glslang’ does not name a type LunarGLASS/Frontends/glslang/GlslangToTopVisitor.cpp:447:19: error: ‘EbvVertexIndex’ is not a member of ‘glslang’ This program is very important for my reasearch. I don't know how to fix this error. Thank you very much!

    opened by SakuraXiaMF 6
Owner
LunarG, Inc.
3D Graphics Software Solutions
LunarG, Inc.
GLSL optimizer based on Mesa's GLSL compiler. Used to be used in Unity for mobile shader optimization.

GLSL optimizer ⚠️ As of mid-2016, the project is unlikely to have any significant developments. At Unity we are moving to a different shader compilati

Aras Pranckevičius 1.6k Oct 2, 2022
⚔️ A tool for cross compiling shaders. Convert between GLSL, HLSL, Metal Shader Language, or older versions of GLSL.

A cross compiler for shader languages. Convert between SPIR-V, GLSL / GLSL ES, HLSL, Metal Shader Language, or older versions of a given language. Cross Shader wraps glslang and SPIRV-Cross, exposing a simpler interface to transpile shaders.

Alain Galvan 171 Sep 27, 2022
The SPIR-V Tools project provides an API and commands for processing SPIR-V modules.

SPIR-V Tools Overview The SPIR-V Tools project provides an API and commands for processing SPIR-V modules. The project includes an assembler, binary m

The Khronos Group 790 Sep 30, 2022
pluggable tool to convert an unrolled TritonAST to LLVM-IR, optimize it and get back to TritonAST

it is fork from https://github.com/fvrmatteo/TritonASTLLVMIRTranslator *WARNINGS: tested only linux(ubuntu 20.04) and only llvm and clang version 10*

pr4gasm 5 Jun 10, 2022
yariv.h is a single C/C++ header to encode and decode SPIR-V shaders into a more compressed form I've called YARI-V.

YARI-V yariv.h is a single C/C++ header to encode and decode SPIR-V shaders into a more compressed form I've called YARI-V. YARI-V is an alternative e

Neil Henning 32 May 3, 2022
C++ front-end package manager for embedding and redistributing with native applications

Pacm Simple C++ package manager Homepage: https://sourcey.com/pacm Documentation: https://sourcey.com/libsourcey/api-pacm/ Dependencies: LibSourcey (b

Sourcey 84 Jun 5, 2022
Projeto pessoal: Obter a temperatura ambiente e através de um termistor ligado a um arduino e disponibilizar esses dados em tempo real via API NodeJS. No front-end os dados são acessados por uma interface em React JS.

INTEGRAÇÃO DA API COM OS DADOS DO ARDUINO FORNECIDOS PELO TERMISTOR Código Desenvolvido por Lucas Muffato. MATERIAIS 1 Placa de Arduino; 1 Cabo de con

Lucas Muffato 35 Aug 16, 2022
Linux System Optimizer and Monitoring - https://oguzhaninan.github.io/Stacer-Web

Linux System Optimizer and Monitoring Reviews Required Packages curl, systemd PPA Repository (for ubuntu) sudo add-apt-repository ppa:oguzhaninan/stac

Oguzhan Inan 8k Oct 4, 2022
Linux Driver for USB WiFi Adapters that are based on the RTL8811AU and RTL8821AU Chipsets

Click for USB WiFi Adapter Information for Linux 8821au ( 8821au.ko ) ?? Linux Driver for USB WiFi Adapters that are based on the RTL8811AU and RTL882

Nick 127 Sep 26, 2022
Linux Driver for USB WiFi Adapters that are based on the RTL8812BU and RTL8822BU Chipsets

Click for USB WiFi Adapter Information for Linux 88x2bu ( 88x2bu.ko ) ?? Linux Driver for USB WiFi Adapters that are based on the RTL8812BU and RTL882

Nick 299 Oct 4, 2022
Linux Driver for USB WiFi Adapters that are based on the RTL8811CU, RTL8821CU and RTL8731AU Chipsets - v5.12.0

Click here for USB WiFi Adapter Information for Linux A FAQ is available at the end of this document. Problem reports go in Issues. Include the inform

Nick 138 Sep 30, 2022
Linux Driver for USB WiFi Adapters that are based on the RTL8812AU Chipset - v5.13.6

Click for USB WiFi Adapter Information for Linux A FAQ is available at the end of this document. Problem reports go in Issues. Problem reports should

Nick 198 Sep 27, 2022
glslcc: Cross-compiler for GLSL shader language (GLSL->HLSL,METAL,GLES,GLSLv3)

glslcc: Cross-compiler for GLSL shader language (GLSL->HLSL,METAL,GLES,GLSLv3) @septag glslcc is a command line tool that converts GLSL code to HLSL,

Sepehr Taghdisian 422 Sep 21, 2022
LoRa Driver for Semtech SX1262 on Linux (PineDio USB Adapter) and BL602 (PineDio Stack BL604)

LoRa Driver for Semtech SX1262 on Linux (PineDio USB Adapter) and BL602 (PineDio Stack BL604) Read the articles... "Build a Linux Driver for PineDio L

Lee Lup Yuen 8 Sep 17, 2022
Firmware for the Boson Ethernet Adapter

Boson Ethernet Firmware This project contains firmware and gateware required to adapt the FLIR boson into an ethernet connection Directory Structure f

Gregory Davill 5 Jul 23, 2022
ControllaBLE - A retro-controllers to Bluetooth BLE adapter

ControllaBLE - A retro-controllers to Bluetooth BLE adapter This is an ESP32 based controller adapter that outputs as a dual joypad through Bluetooth

null 13 Dec 1, 2021
Nissan Pathfinder R51 Head Unit Adapter

Nissan Pathfinder R51 Head Unit Adapter This repo contains documentation and code to turn an Arduino into a CAN Bus climate control adapter for the Ni

Ryan Bourgeois 1 Apr 19, 2022
ESP based wifi adapter for Celestron telescopes

CelestronESPWifi ESP8266 based wifi adapter for Celestron telescopes The official Celestron Skyportal WiFi module is €190~. (aka SkyQ Link, aka Skylin

null 7 Jun 25, 2022
MQB steering wheel adapter to retrofit into PQ46 platform (VW Passat b6/b7, CC)

MQB steering wheel adapter (retrofit MQB steering wheel into PQ46 platform - VW Passat b6/b7, CC) The adapter supports on-wheel buttons (media and cru

null 13 Sep 11, 2022