HLSL shader prototype with Vulkan

As part of the Vulkan prototype, I'm experimenting with compiling HLSL shaders to SPIR-V.

The Vulkan API doesn't have a high level shader language attached. Instead, it works with a new intermediate bytecode format, called SPIR-V. This works with the LLVM compiler system, so in theory we plug in different front ends to allow various high level languages to compile to SPIR-V.

That sounds great for the future... But right now, there doesn't seem to be a great path for generating the SPIR-V bytecode.

All of the XLE shaders are HLSL... So how do we use them with Vulkan? Let's consider the options.

Preprocessor framework

One option is to convert the shader code to GLSL using a preprocessor framework. HLSL and GLSL are closely related... But they vary significantly in the way the shader interface is declared (eg, shader resources, vertex elements, built in values, etc).

We can get around this by using a system of preprocessor macros and complex #includes. We would end up with native HLSL and GLSL code that does the same thing. The core shader code that does the math and lighting (etc) should be fairly compatible between languages... It's just the interface that is an issue.

However this kind of approach it's a bit awkward, and difficult maintain in the long term. And it might mean dropping support for some of the more unusual D3D features I'm using (such as for dynamic linking).

Also, it looks like GLSL might not be around a very long time. It could possibly go the way of OpenGL in the medium term. So it doesn't make sense to invest a lot of time into GLSL, just to have to be replaced with something else later.

Cross compile

Another option is to try to convert the HLSL output to SPIR-V by way of a cross compiler.

There is an interesting project here https://github.com/James-Jones/HLSLCrossCompiler. This will take as input HLSL bytecode, and output GLSL high level shader code.

HLSL bytecode is a much simpler than HLSL source (given that it's mostly assembler like instructions). So this approach should be more maintainable.

The issue here is there is no standard equivalent bytecode form of GLSL. The output from the cross compile is just high level GLSL code.

Once we have the GLSL code, we can generate SPIR-V using the GLSL pipeline in the Vulkan SDK.

Cross compile problems

The process for compiling a shader becomes quite long:

1. compile HLSL to HLSL bytecode using D3DCompiler_47.dll (ie, just like D3D11)
2. translate HLSL bytecode to GLSL
3. parse GLSL to AST
4. translate GLSL AST to SPIR-V

The big problem here is that each step adds it's own restrictions. So in the end we up with the sum of all those restrictions.

Another issue is that a few of these steps have know bugs and version support issues. The HLSLCrossCompiler seems to translate several instruction incorrectly. And the GLSL AST to SPIR-V seems to currently only support a subset of GLSL.

For example, the [earlydepthstencil] annotations in HLSL get correctly converted into GLSL -- but then the SPIR-V translator doesn't support them!

Also, GLSL to SPIR-V doesn't fully support GLSL version 4 shaders yet... So I'm using GLSL 3.3. But that may not be able to support all of the features I'm using in HLSL.

Fortunately, both the HLSL cross compile and the GLSL to SPIR-V compiler are open-source... So there is room to make fixes and improvements if necessary. And it means those projects will be more open about what is working, and what isn't.

Prototype results

It seems that the cross-compile approach is actually going to be best, despite the problems. So, I've got an early version of this process working in the "experimental" branch.

In this branch, there is a new implementation of RenderCore::ShaderService::ILowLevelCompiler that does HLSL -> SPIR-V translation.

It requires some libraries from the Vulkan SDK currently... I'm not sure how I will incorporate those into linking process (perhaps it will mean a new VULKAN_SDK environment variable).

I've only tested a few shaders so far.... But it works! It actually does compile, and we end up with SPIR-V at the end. Wow, pretty incredible.

I've added a new "VULKAN=1" shader define. This will be probably be required for enabling and disabling certain shader features for this pipeline.

The next step will be working on how to best extract reflection information from the shaders. We've actually got many ways to do that now... But I want to check to make sure we can still get the information we need out.