Like many of you, when I work on a graphics project I sometimes have a need to compile some shaders. Usually, I’m writing in C++ using Visual Studio, and I’d like to get my shaders built using the same workflow as the rest of my code. Visual Studio these days has built-in support for HLSL via fxc, but what if we want to use the next-gen dxc compiler?

This post is a how-to for adding support for a custom toolchain—such as dxc, or any other command-line-invokable tool—to a Visual Studio project, by scripting MSBuild (the underlying build system Visual Studio uses). We won’t quite make it to parity with a natively integrated language, but we’re going to get as close as we can.

If you don’t want to read all the explanation but just want some working code to look at, jump down to the Example Project section.

This article is written against Visual Studio 2017, but it may also work in some earlier VSes (I haven’t tested).

MSBuild

Before we begin, it’s important you understand what we’re getting into. Not to mince words, but MSBuild is a stringly typed, semi-documented, XML-guzzling, paradigmatically muddled, cursed hellmaze. However, it does ship with Visual Studio, so if you can use it for your custom build steps, then you don’t need to deal with any extra add-ins or software installs.

To be fair, MSBuild is open-source on GitHub, so at least in principle you can dive into it and see what the cursed hellmaze is doing. However, I’ll warn you up front that many of the most interesting parts vis-à-vis Visual Studio integration are not included in the Git repo, but are hidden away in VS’s build extension DLLs. (More about that later.)

My jumping-off point for this enterprise was this blog post by Mike Nicolella. Mike showed how to set up an MSBuild .targets file to create an association between a specific file extension in your project, and a build rule (“target”, in MSBuild parlance) to process those files. We’ll review how that works, then extend it and jazz it up a bit to get some more quality-of-life features.

MSBuild docs (such as they are) can be found on MSDN here. Some more information can be gleaned by looking at the C++ build rules installed with Visual Studio; on my machine they’re in C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets. For example, the file Microsoft.CppCommon.targets in that directory contains most of the target definitions for C++ compilation, linking, resources and manifests, and so on.

Adding A Custom Target

As shown in Mike’s blog post, we can define our own build rule using a couple of XML files which will be imported into the VS project. (I’ll keep using shader compilation with dxc as my running example, but this approach can be adapted for a lot of other things, too.)

First, create a file dxc.targets—in your project directory, or really anywhere—containing the following:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup>
    <!-- Include definitions from dxc.xml, which defines the DXCShader item. -->
    <PropertyPageSchema Include="$(MSBuildThisFileDirectory)dxc.xml" />
    <!-- Hook up DXCShader items to be built by the DXC target. -->
    <AvailableItemName Include="DXCShader">
      <Targets>DXC</Targets>
    </AvailableItemName>
  </ItemGroup>

  <Target
    Name="DXC"
    Condition="'@(DXCShader)' != ''"
    BeforeTargets="ClCompile">
    <Message Importance="High" Text="Building shaders!!!" />
  </Target>
</Project>

And another file dxc.xml containing:

<?xml version="1.0" encoding="utf-8"?>
<ProjectSchemaDefinitions xmlns="http://schemas.microsoft.com/build/2009/properties">
  <!-- Associate DXCShader item type with .hlsl files -->
  <ItemType Name="DXCShader" DisplayName="DXC Shader" />
  <ContentType Name="DXCShader" ItemType="DXCShader" DisplayName="DXC Shader" />
  <FileExtension Name=".hlsl" ContentType="DXCShader" />
</ProjectSchemaDefinitions>

Let’s pause for a moment and take stock of what’s going on here. First, we’re creating a new “item type”, called DXCShader, and associating it with the extension .hlsl. That way, any files we add to our project with that extension will automatically have this item type applied.

Second, we’re instructing MSBuild that DXCShader items are to be built with the DXC target, and we’re defining what that target does. For now, all it does is print a message in the build output, but we’ll get it doing some actual work shortly.

A few miscellaneous syntax notes:

• Yes, you need two separate files. No, there’s no way to combine them, AFAICT. This is just the way MSBuild works.
• The syntax @(DXCShader) means “the list of all DXCShader items in the project”. The Condition attribute on a target says under what conditions that target should execute: if the condition is false, the target is skipped. Here, we’re executing the target if the list @(DXCShader) is non-empty.
• BeforeTargets="ClCompile" means this target will run before the ClCompile target, i.e. before C/C++ source files are compiled with cl.exe. This is because we’re going to output our shader bytecode to headers which will get included into C++, so the shader compile step needs to run earlier.
• Importance="High" is needed on the <Message> task for it to show up in the VS IDE on the default verbosity setting. Lower importances will be masked unless you turn up the verbosity.

To get this into your project, in the VS IDE right-click the project → Build Dependencies… → Build Customizations, then click “Find Existing” and point it at dxc.targets. Alternatively, add this line to your .vcxproj (as a child of the root <Project> element, doesn’t matter where):

<Import Project="dxc.targets" />

Now, if you add a .hlsl file to your project it should automatically show up as type “DXC Shader” in the properties; and when you build, you should see the message Building shaders!!! in the output.

Incidentally, in dxc.xml you can also set up property pages that will show up in the VS IDE on DXCShader-type files. This lets you define your own metadata and let users configure it per file. I haven’t done this, but for example, you could have properties to indicate which shader stages or profiles the file should be compiled for. The <Target> element can then have logic that refers to those properties. Many examples of the XML to define property pages can be found in C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\1033 (or a corresponding location depending on which version of VS you have). For example, custom_build_tool.xml in that directory defines the properties for the built-in Custom Build Tool item type.

Invoking The Tool

Okay, now it’s time to get our custom target to actually do something. Mike’s blog post used the MSBuild <Exec> task to run a command on each source file. However, we’re going to take a different tack and use the Visual Studio <CustomBuild> task instead.

The <CustomBuild> task is the same one that ends up getting executed if you manually set your files to “Custom Build Tool” and fill in the command/inputs/outputs metadata in the property pages. But instead of putting that in by hand, we’re going to set up our target to generate the metadata and then pass it in to <CustomBuild>. Doing it this way is going to let us access a couple handy features later that we wouldn’t get with the plain <Exec> task.

Add this inside the DXC <Target> element:

<!-- Setup metadata for custom build tool -->
<ItemGroup>
  <DXCShader>
    <Message>%(Filename)%(Extension)</Message>
    <Command>
      "$(WDKBinRoot)\x86\dxc.exe" -T vs_6_0 -E vs_main %(Identity) -Fh %(Filename).vs.h -Vn %(Filename)_vs
      "$(WDKBinRoot)\x86\dxc.exe" -T ps_6_0 -E ps_main %(Identity) -Fh %(Filename).ps.h -Vn %(Filename)_ps
    </Command>
    <Outputs>%(Filename).vs.h;%(Filename).ps.h</Outputs>
  </DXCShader>
</ItemGroup>

<!-- Compile by forwarding to the Custom Build Tool infrastructure -->
<CustomBuild Sources="@(DXCShader)" />

Now, given some valid HLSL source files in the project, this will invoke dxc.exe twice on each one—first compiling a vertex shader, then a pixel shader. The bytecode will be output as C arrays in header files (-Fh option). I’ve just put the output headers in the main project directory, but in production you’d probably want to put them in a subdirectory somewhere.

Let’s back up and look at the syntax in this snippet. First, the <ItemGroup><DXCShader> combo basically says “iterate over the DXCShader items”, i.e. the HLSL source files in the project. Then what we’re doing is adding metadata: each of the child elements—<Message>, <Command>, and <Outputs>—becomes a metadata key/value pair attached to a DXCShader.

The %(Foo) syntax accesses item metadata (within a previously established context for “which item”, which is here created by the iteration over the shaders). All MSBuild items have certain built-in metadata like path, filename, and extension; we’re building on those to construct additional metadata, in the format expected by the <CustomBuild> task. (It matches the metadata that would be created if you set up the command line etc. manually in the Custom Build Tool property pages.)

Incidentally, the $(WDKBinRoot) variable (“property”, in MSBuild-ese) is the path to the Windows SDK bin folder, where lots of tools like dxc live. It needs to be quoted because it can (and usually does) contain spaces. You can find out these things by running MSBuild with “diagnostic” verbosity (in VS, go to Tools → Options → Projects and Solutions → Build and Run → “MSBuild project build output verbosity”)—this will spit out all the defined properties plus a ton of logging about which targets are running and what they’re doing.

Finally, after setting up all the required metadata, we simply pass it to the <CustomBuild> task. (This task isn’t part of core MSBuild, but is defined in Microsoft.Build.CPPTasks.Common.dll—an extension plugin to MSBuild that comes with Visual Studio.) Again we see the @(DXCShader) syntax, meaning to pass in the list of all DXCShader items in the project. Internally, <CustomBuild> iterates over it and invokes your specified command lines.

Incremental Builds

At this point, we have a working custom build! We can simply add .hlsl files to our project, and they’ll automatically be compiled by dxc as part of the build process, without us having to do anything. Hurrah!

However, while working with this setup you will notice a couple of problems.

1. When you modify an HLSL source file, Visual Studio will not reliably detect that it needs to recompile it. If the project was up-to-date before, hitting Build will do nothing! However, if you have also modified something else (such as a C++ source file), then the build will pick up the shaders in addition.
2. Anytime anything else gets built, all the shaders get built. In other words, MSBuild doesn’t yet understand that if an individual shader is already up-to-date then it can be skipped.

Fortunately, we can easily fix these. But first, why are these problems happening at all?

VS and MSBuild depend on .tlog (tracker log) files to cache information about source file dependencies and efficiently determine whether a build is up-to-date. Somewhere inside your build output directory there will be a folder full of these logs, listing what source files have gotten built, what inputs they depended on (e.g. headers), and what outputs they generated (e.g. object files). The problem is that our custom target isn’t producing any .tlogs.

Conveniently for us, the <CustomBuild> task supports .tlog handling right out of the box; we just have to turn it on! Change the <CustomBuild> invocation in the targets file to this:

<!-- Compile by forwarding to the Custom Build Tool infrastructure,
     so it will take care of .tlogs -->
<CustomBuild
  Sources="@(DXCShader)"
  MinimalRebuildFromTracking="true"
  TrackerLogDirectory="$(TLogLocation)" />

That’s all there is to it—now, modified HLSL files will be properly detected and rebuilt, and unmodified ones will be properly detected and not rebuilt. This also takes care of deleting the previous output files when you do a clean build. This is one reason to prefer using the <CustomBuild> task rather than the simpler <Exec> task (we’ll see another reason a bit later).

Thanks to Olga Arkhipova at Microsoft for helping me figure out this part!

Header Dependencies

Now that we have dependencies hooked up for our custom toolchain, a logical next step is to look into how we can specify extra input dependencies—so that our shaders can have #includes, for example, and modifications to the headers will automatically trigger rebuilds properly.

The good news is that yes, we can do this by adding an <AdditionalInputs> metadata key to our DXCShader items. Files listed there will get registered as inputs in the .tlog, and the build system will do the rest. The bad news is that there doesn’t seem to be an easy way to detect on a file-by-file level which additional inputs are needed.

This is frustrating because Visual Studio actually includes a utility for tracking file accesses in an external tool! It’s called tracker.exe and lives somewhere in your VS installation. You give it a command line, and it’ll detect all files opened for reading by the launched process (presumably by injecting a DLL and detouring CreateFile(), or something along those lines). I believe this is what VS uses internally to track #includes for C++—and it would be perfect if we could get access to the same functionality for custom toolchains as well.

Unfortunately, the <CustomBuild> task explicitly disables this tracking functionality. I was able to find this out by using ILSpy to decompile the Microsoft.Build.CPPTasks.Common.dll. It’s a .NET assembly, so it decompiles pretty cleanly, and you can examine the innards of the CustomBuild class. It contains this snippet, in the ExecuteTool() method:

bool trackFileAccess = base.TrackFileAccess;
base.TrackFileAccess = false;
num = base.TrackerExecuteTool(pathToTool2, responseFileCommands, commandLineCommands);
base.TrackFileAccess = trackFileAccess;

That is, it’s turning off file access tracking before calling the base class method that would otherwise invoke the tracker. I’m sure there’s a reason why they did that, but sadly it’s stymied my attempts to get automatic #include tracking to work for shaders.

(We could also invoke tracker.exe manually in our command line, but then we face the problem of merging the tracker-generated .tlog into that of the <CustomBuild> task. They’re just text files, so it’s potentially doable…but that is way more programming than I’m prepared to attempt in an XML-based scripting language.)

Although we can’t get fine-grained file-by-file header dependencies, we can still set up conservative dependencies by making every HLSL source file depend on every header. This will result in rebuilding all the shaders whenever any header is modified—but better to rebuild too much than not enough. We can find all the headers using a wildcard pattern and an <ItemGroup>. Add this to the DXC <Target>, before the “setup metadata” section:

<!-- Find all shader headers (.hlsli files) -->
<ItemGroup>
  <ShaderHeader Include="*.hlsli" />
</ItemGroup>
<PropertyGroup>
  <ShaderHeaders>@(ShaderHeader)</ShaderHeaders>
</PropertyGroup>

You could also set this to find .h files under a Shaders subdirectory, or whatever you prefer. The ** wildcard is available for recursively searching subdirectories, too.

Then add this inside the <ItemGroup><DXCShader> section:

<AdditionalInputs>$(ShaderHeaders)</AdditionalInputs>

We have to do a little dance here, first forming the ShaderHeader item list, then expanding it into the ShaderHeaders property, and finally referencing that in the metadata. I’m not sure why, but if I try to use @(ShaderHeader) directly in the metadata it just comes out blank. Perhaps it’s not allowed to have nested iteration over item lists in MSBuild.

In any case, after making these changes and rebuilding, the build should now pick up any changes to shader headers. Woohoo!

Error/Warning Parsing

There’s just one more bit of sparkle we can easily add. When you compile C++ and you get an error or warning, the VS IDE recognizes it and produces a clickable link that takes you to the source location. If a custom build step emits error messages in the same format, they’ll be picked up as well—but what if your custom toolchain has a different format?

The dxc compiler emits errors and warnings in gcc/clang format, looking something like this:

Shader.hlsl:12:15: error: cannot convert from 'float3' to 'float4'

It turns out that Visual Studio already does recognize this format (at least as of version 15.9), which is great! But if it didn’t, or in case you’ve got a tool with some other message format, it turns out you can provide a regular expression to find errors and warnings in the tool output. The regex can even supply source file/line information, and the errors will become clickable in the IDE, just as with C++. (This is all totally undocumented and I only know about it because I spotted the code while browsing through the decompiled CPPTasks DLL. If you want to take a look for yourself, the juicy bit is the VCToolTask.ParseLine() method.)

This will use .NET regex syntax, and in particular, expects a certain set of named captures to provide metadata. By way of example, here’s the regex I wrote for gcc/clang-format errors:

(?'FILENAME'.+):(?'LINE'\d+):(?'COLUMN'\d+): (?'CATEGORY'error|warning): (?'TEXT'.*)

FILENAME, LINE, etc. are the names the parsing code expects for the metadata. There’s one more I didn’t use: CODE, for an error code (like C2440, etc.). The only required one is CATEGORY, without which the message won’t be clickable (and it must be one of the words “error”, “warning”, or “note”); all the others are optional.

To use it, pass the regex to the <CustomBuild> task like so:

<CustomBuild
  Sources="@(DXCShader)"
  MinimalRebuildFromTracking="true"
  TrackerLogDirectory="$(TLogLocation)"
  ErrorListRegex="(?'FILENAME'.+):(?'LINE'\d+):(?'COLUMN'\d+): (?'CATEGORY'error|warning): (?'TEXT'.*)" />

Example Project

Here’s a complete VS2017 project with all the features we’ve discussed, a couple demo shaders, and a C++ file that includes the compiled bytecode (just to show that works).

Download Example Project (.zip, 4.3 KB)

And for completeness, here’s the final contents of dxc.targets:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup>
    <!-- Include definitions from dxc.xml, which defines the DXCShader item. -->
    <PropertyPageSchema Include="$(MSBuildThisFileDirectory)dxc.xml" />
    <!-- Hook up DXCShader items to be built by the DXC target. -->
    <AvailableItemName Include="DXCShader">
      <Targets>DXC</Targets>
    </AvailableItemName>
  </ItemGroup>

  <Target
    Name="DXC"
    Condition="'@(DXCShader)' != ''"
    BeforeTargets="ClCompile">

    <Message Importance="High" Text="Building shaders!!!" />

    <!-- Find all shader headers (.hlsli files) -->
    <ItemGroup>
      <ShaderHeader Include="*.hlsli" />
    </ItemGroup>
    <PropertyGroup>
      <ShaderHeaders>@(ShaderHeader)</ShaderHeaders>
    </PropertyGroup>

    <!-- Setup metadata for custom build tool -->
    <ItemGroup>
      <DXCShader>
        <Message>%(Filename)%(Extension)</Message>
        <Command>
          "$(WDKBinRoot)\x86\dxc.exe" -T vs_6_0 -E vs_main %(Identity) -Fh %(Filename).vs.h -Vn %(Filename)_vs
          "$(WDKBinRoot)\x86\dxc.exe" -T ps_6_0 -E ps_main %(Identity) -Fh %(Filename).ps.h -Vn %(Filename)_ps
        </Command>
        <AdditionalInputs>$(ShaderHeaders)</AdditionalInputs>
        <Outputs>%(Filename).vs.h;%(Filename).ps.h</Outputs>
      </DXCShader>
    </ItemGroup>

    <!-- Compile by forwarding to the Custom Build Tool infrastructure,
         so it will take care of .tlogs and error/warning parsing -->
    <CustomBuild
      Sources="@(DXCShader)"
      MinimalRebuildFromTracking="true"
      TrackerLogDirectory="$(TLogLocation)"
      ErrorListRegex="(?'FILENAME'.+):(?'LINE'\d+):(?'COLUMN'\d+): (?'CATEGORY'error|warning): (?'TEXT'.*)" />
  </Target>
</Project>

The Next Level

At this point, we have a pretty usable MSBuild customization for compiling shaders, or using other kinds of custom toolchains! I’m pretty happy with it. However, there’s still a couple of areas for improvement.

• As mentioned before, I’d like to get file access tracking to work so we can have exact dependencies for included files, rather than conservative (overly broad) dependencies.
• I haven’t done anything with parallel building. Currently, <CustomBuild> tasks are run one at a time. There is a <ParallelCustomBuild> task in the CPPTasks assembly…unfortunately, it doesn’t support .tlog updating or the error/warning regex, so it’s not directly usable here.

To obtain these features, I think I’d need to write my own build extension in C#, defining a custom task and calling it in place of <CustomBuild> in the targets file. It might not be too hard to get that working, but I haven’t attempted it yet.

In the meantime, now that the hard work of circumventing the weird gotchas and reverse-engineering the undocumented innards has been done, it should be pretty easy to adapt this .targets setup to other needs for code generation or external tools, and have them act mostly like first-class citizens in our Visual Studio builds. Cheers!