Processing textures with the XLE scriptable asset path

I've included a few Gradle scripts in the XLE distribution. This is mostly just a simple set of tools I use for my own testing. But you may find it useful for your own needs.

Of course this system isn't designed to be 100% robust and fool proof. Actually, it's just a few simple scripts. But it is scalable and flexible.

Gradle

At heart, an "asset path" is some system that can identify changed assets, recognize processing working that needs to be performed with those assets, and schedule that work.

For example, we might have an asset for a sky background texture. Various processing tools need to pre-filter this texture so it can be used for image based lighting. Whenever the texture changes (or when the processing tools change) we want to execute the processing steps and produce intermediate assets.

XLE does some processing at runtime. But that is only practical for short processing steps. Expensive processing steps need some other solution.

So we need some build system to manage assets and dependences. Are requirements are similar to build systems we use for code. But most code-oriented build systems don't work well for assets.

I picked Gradle because of it's procedural nature. It allows us to specify input assets -- but also to give instructions to the build path on how to handle that object. For example, we need to tell the build path if a texture is a sky texture, a normals texture, or some other type... That kind of thing just falls out of Gradle very easily.

Executing Gradle

First, you need to install Gradle, from: http://gradle.org/gradle-download/

In Tools/AssetPath, we have the "settings.gradle" root project file. Also, this folder contains some "groovy" source files that contain gradle task type implementations. So, there is a task type for processing sky textures -- which basically just involves executing a number of command line operations.

I've also defined a root task called "tex". This is where gradle starts to shine. The "tex" task searches through the entire working folder for build.gradle files that contain tasks whose names being with "tex...". All of the tasks found become subtasks of the root "tex" task.

So, we can create a "build.gradle" in any folder in the working directory. And if we add tasks to that file that being with "tex" (typically "tex0", "tex1", etc), they will be automatically added as subtasks of the root "tex" task.

This is important because when we execute gradle (using the "Tools/AssetPath/execute.bat" batch file) and pass the root task "tex" on the command line, this has the effect of execute all "tex..." tasks in the entire working directory.

Pretty cool, right? We can also execute individual tasks using the normal gradle command line.

There is an example in git in "Working/Game/xleres/DefaultResources/build.gradle." This project will generate the standard lookup tables.

Processing Textures

Executing Gradle is just the first step. Gradle is just going to chain together command line operations. Those operations require a bunch of tools.

One of the most useful task types is "xle.EquiRectEnv." This takes in a equirectangular (ie, paranoramic) environment map and produces 3 important textures: Cubemap background, Specular IBL texture and Diffuse IBL texture.

This get a little complicated. This is just the pipeline I use for processing these textures myself. So it's a little complicated and involved right now.

First, install these:

1. AMD CubemapGen (http://developer.amd.com/tools-and-sdks/archive/legacy-cpu-gpu-tools/cubemapgen/)
2. Sébastien Lagarde's modified CubemapGen: https://seblagarde.wordpress.com/2012/06/10/amd-cubemapgen-for-physically-based-rendering/
3. nvcompress ** from nvidia-texture-tools (see below)
4. TextureProcess sample from XLE

All of these must in the system environment variable path. In the case of TextureProcess, don't move the executable from the Finals_** folders (because it needs to find the working directory for shaders).

I use a slightly slightly modified version of nvidia-texture-tools that works better with HDR textures. The standard nvidia-texture-tools always tonemaps HDR textures on load-in (frustratingly). You can find my modified version here: https://github.com/djewsbury/nvidia-texture-tools. This modified version allows us to read HDR files (from .hdr and other formats) and write unprocessed floating point .dds file.

Sébastien Lagarde's ModifiedCubeMapGen is used to generate the diffuse IBL texture. This goes via a spherical harmonic representation before arriving at a small cubemap.

XLE's TextureProcess.exe is used to generate the specular IBL texture, as well as compressing to BC6. The specular IBL process reads from a equirectangular HDR map and writes a cubemap.

All of these textures need to be compress to BC6. Awkwardly, nvidia-texture-tools doesn't support compressing HDR data to BC6 properly, but TextureProcess.exe can do that via the DirectXTex library.

So, once you've got all of that, compiled it all, put in all in the path... Then you're ready to process textures.

Intermediate outputs from this process get written into the "int/u" (or intermediate/universal) directory. From there, you can copy it out, or do what you like. You should get a (file).dds, (file)_diffuse.dds & (file)_specular.dds.

Make it your own!

It's difficult to get this working the first time. There's a lot of work. But once it's going, all of the parts are extensible and exchangeable. It's a simple, but very flexible system. Handy for managing small, everyday processing tasks.