DreamShaderLang

DreamShader » C++ API » DreamShaderLang

DreamShaderLang is the front end and the middle end: the text goes in, a lowered graph comes out — and the other way round. Lexer, preprocessor, both parsers (2.0 and the 1.x legacy front end), AST, printer, binder, the graph IR with its passes, validator and comparator, the writer that turns an IR back into source, the 1.x → 2.0 migrator, and the diagnostic types live here — and nothing else does. It was split out of DreamShader for 2.0 so the language can be exercised without an editor, reused by a language service, and reasoned about as a language rather than as a plugin.

The line the module does not cross is the engine: an FIRModule names no UObject, no asset and no expression class object. Engine knowledge arrives as plain data in an FBuiltinCatalog, which the Editor module fills by reflection and can also save to, and load from, JSON.

The Core-only rule

DreamShaderLang.Build.cs declares exactly one dependency:

PublicDependencyModuleNames.AddRange(new[] { "Core" });

This is a hard boundary, not a current state of affairs:

• No UObject, no Engine, no CoreUObject. Nothing in the module can name a UMaterial, a UMaterialExpression or an asset path type.
• No IPluginManager, no GetDefault<>(), no config access. The module cannot look anything up about the project it is running in.
• Engine facts arrive as parameters. The DS_* preprocessor constants, asset roots and expression class names are supplied by the host as plain data — DreamShaderDefineResolution.h in the DreamShader module is the half that builds a define table by asking the engine; DreamShaderLang only consumes one.
• Every header compiles in a commandlet, a test, or a future standalone language server. The front end creates no objects and touches no files: FLangSourceText is handed text that somebody else read.

Public headers

Lang/ — the 2.0 front end

Everything here is in UE::DreamShader::Lang. Include paths keep the Lang/ prefix, e.g. #include "Lang/LangParser.h".

Semantic/ — the binder

UE::DreamShader::Lang, one header. Binding is the step between "it parses" and "it can be lowered": every name resolved, every expression typed, every call matched against a signature or the catalog, and the file's products decided.

A bound module keeps pointers into the FModule it bound and into whatever the include resolver returned; both have to outlive it. Nothing downstream of the builder does — an FIRModule points back at neither.

IR/ — the graph IR

UE::DreamShader::IR, twelve headers. The IR is the material graph before it is a material graph: nodes, operands, properties and products, with no engine type anywhere in it. Include paths keep the IR/ prefix.

The IR holds no source positions in its identity: FIRSourceRef rides on a node for tooling (the asset's DreamShader.SourceSpans table, node ↔ source navigation) and is never part of a dedupe key. Two structurally identical nodes merge no matter where they were written.

Decompile/ and Migrate/ — the way back

UE::DreamShader::Lang, one header each. Neither knows about assets: the editor module imports a graph into an FIRModule and hands it over.

Module root — what moved here in 2.0

Three of the four headers below were in the DreamShader module through 1.9.x and moved into DreamShaderLang for 2.0. They kept their UE::DreamShader namespace and their include paths, so existing callers need no source change — only a DreamShaderLang entry in their Build.cs dependency list. DreamShaderLangModule.h is new in 2.0.

FDreamShaderError (1.x, FString/FText message) and FLangDiagnostic (2.0, code + severity + FText + span) coexist on purpose: the 1.x pipeline still carries the former, and the 2.0 front end never produces one. They are not converted implicitly.

Entry points

The module is a pipeline of six calls — preprocess, lex, parse, bind, lower, run the passes — plus a validator, two dumps, a printer, and one call for parsing a single expression on its own.

// MyTooling.Build.cs:  PrivateDependencyModuleNames.AddRange(new[] { "Core", "DreamShaderLang" });

#include "Lang/LangParser.h"
#include "Lang/LangPrinter.h"

using namespace UE::DreamShader::Lang;

void Normalise(const FString& Path, const FString& Text)
{
    const FLangSourceText Source(Path, Text);
    const FLangParseResult Result = ParseDreamShaderLang(Source);

    for (const FLangDiagnostic& Diagnostic : Result.Diagnostics.GetDiagnostics())
    {
        UE_LOG(LogTemp, Warning, TEXT("%s(%d,%d): %s"),
            *Diagnostic.FilePath, Diagnostic.Span.Line, Diagnostic.Span.Column,
            *FLangDiagnosticSink::ToWireString(Diagnostic));
    }

    if (Result.Succeeded())
    {
        const FString Printed = PrintDreamShaderLang(*Result.Module);
    }
}

Front-end selection

FLangParseOptions::Frontend defaults to ELangFrontend::Auto, which decides from the file extension carried by FLangSourceText::GetPath():

The legacy front end lowers 1.x constructs onto the same AST node kinds — a Shader block becomes uniforms, a #pragma material and one exported function — so nothing downstream has to know which parser produced a module. What the 1.x text said beyond that (block names, sections, renames, output selections) rides beside the tree in FLangParseResult::Legacy, and the declarations carry bLegacy, which is what makes the binder apply the documented 1.x rules to them and to nothing else.

Printer contract

The printer's promise is structural fidelity, not textual: parse(print(parse(X))) yields the same tree as parse(X). Concretely — parentheses are emitted where FParenExpr recorded them and wherever precedence requires; /// blocks are re-emitted one directive per line after the free text; an opaque (/// @custom) body is written verbatim from RawBody; pragmas and includes keep their spelling; literals keep the lexeme as written, so 1.0f, 0x10 and 2u survive. Comments other than /// are reproduced when the parse kept them: with FLangParseOptions::bKeepTrivia every comment and blank line is attached to the declaration or statement it stood by (FModule::Trivia), and the printer writes them back — which is what lets dsc migrate promise that no comment is lost. Without the option they are dropped, as before.

FLangPrintOptions::NewLine defaults to \n, but an opaque body is a slice of its source and keeps that file's terminators, so printing a CRLF file yields mixed terminators. That is stable: the next parse slices the same bytes back out, so the fixed point still holds.

Opaque bodies

A function whose /// block carries @custom has its body captured verbatim: FFunctionDecl::Body is null, bOpaqueBody is true, RawBody is the text between the braces and BodySpan covers the braces inclusive. The parser reads only the braces — they must balance, and strings, comments and # lines inside cannot unbalance them because the lexer already made each one a single token.

Because the text is not DreamShaderLang, ParseDreamShaderLang discards every lexical diagnostic whose span starts inside a BodySpan. A $, a @, a ' or a mid-line # in there is not a fact about the program, and the shader compiler is the one that gets to judge it. The suppression is scoped to those spans: the same character one line outside is still DSH2101 / DSH2106.

Diagnostics

Every raise site names its code literally, so .skill/gen-diagnostics.ps1 can find it:

Diagnostics.Error(TEXT("DSH2151"), Span, LOCTEXT("ExpectedExpression", "expected an expression"));

The messages are LOCTEXT under a per-file LOCTEXT_NAMESPACE of the form "DreamShader.Lang.<Unit>", so the editor can localise them. Assert on the code, never on the message text.

The binder, the builder, the passes, the validator and the custom-HLSL builder raise through the same sink, with their own ranges:

DSH1030–DSH1042 (the preprocessor's own codes) moved into this module with the preprocessor and kept their numbers. The prose for every code lives under Diagnostics.

Testing: the Lang corpus

The front end is covered by a data-driven corpus, not by hand-written per-feature tests. Fixtures live under Tests/Corpus/Lang/, one .dss or .dsh per feature, each optionally paired with a <name>.expected.json golden; a complex automation test enumerates the tree at runtime and surfaces every fixture as its own sub-test.

Each case is parsed once through ParseDreamShaderLang, then checked against its golden:

• outcome — ok or error. Without a golden the default is "a positive fixture parses, a *.bad.* fixture fails".
• errorContains / warningsContain — substrings (case-sensitive) of FLangDiagnosticSink::ToWireString of each diagnostic of that severity. In practice: the DSHnnnn code.
• module — structural counts over the parsed FModule: declarations (pragmas and includes count), pragmas, includes, structs, uniforms (Uniform storage only), constants (StaticConst only), functions (all three linkages), exports, externs, opaqueBodies, entry (the first function whose signature is void (inout material), regardless of linkage) and roundtrip.
• roundtrip — print → parse → print is byte-identical, which is the printer's contract expressed as a test.

Adding coverage for a new construct is dropping a file in the tree; no C++ changes, no recompile. Run the whole layer headlessly:

"F:\UnrealEngine\UE_Moon\Engine\Binaries\Win64\UnrealEditor-Cmd.exe" ^
  "I:\UnrealProject_Moon\DEV_58\MoonEngineSample\MoonEngineSample.uproject" ^
  -ExecCmds="Automation RunTests DreamShader.Lang2; Quit" ^
  -nullrhi -unattended -nopause -nosplash -log

Adding -DreamShaderUpdateGolden rewrites every golden from the actual result instead of asserting. Review the diff by hand before committing — that switch is how a regression becomes a baseline.

Alongside the corpus, each front-end unit has its own focused test file (DreamShaderLangLexerTests.cpp, …DeclarationTests.cpp, …ExpressionTests.cpp, …PrinterTests.cpp), all under DreamShader.Lang2.*.

See also

• What the 2.0 pipeline does — the language surface and the pipeline's limits, from the author's side
• C++ API index — the other four modules
• Custom Pass pipelines — the .dsp language, from the author's side
• Preprocessor — #if and the define table, documented from the language side
• Diagnostics index — every DSHnnnn with cause and fix
• Testing — the automation suite and the fixture corpus