Testing

DreamShader » Contributing » Testing

The plugin's Unreal automation suite, and the on-disk fixture corpus that thirteen of its tests enumerate at run time.

Synopsis

& "<EngineDir>\Engine\Binaries\Win64\UnrealEditor-Cmd.exe" --% "<ProjectDir>\<Project>.uproject" -ExecCmds="Automation RunTests <filter>; Quit" -nullrhi -unattended -nopause -nosplash -log

<filter> is any prefix of a test name; DreamShader runs everything. --% is PowerShell's stop-parsing token: everything after it is handed to the executable verbatim, so the quotes around -ExecCmds= survive. The same line without --% is what you paste into cmd.exe.

Optional switches: -nullrhi, -DreamShaderUpdateGolden, -NoDreamShaderEditorBridge. See Command-line switches.

Test counts

The complex runners enumerate the corpus tree at run time, so the runnable-test count moves with the fixture count and no C++ changes when a fixture is added. DreamShader.Lang2.RoundtripIR sweeps fixtures that belong to other layers, which is why the runnable count is not "simple + fixtures".

Test groups

The split the source records: the fast DreamShader.Lang.* and DreamShader.Lang2.* layers gate pull requests, the slow DreamShader.Compiler* and DreamShader.Gen.* layers run nightly. The whole suite takes about a minute and a half headlessly.

[!WARNING] Close the editor first. A run compiles sources and writes assets under /Game/DreamShaderTests; with an editor open on the same project the two fight over the bridge's write ownership and over package files.

Running headlessly

The commandlet driver's ownership-aware cleanup can be tested without Unreal:

pwsh -NoProfile -File Tools/Tests/Test-DscAssetCleanup.ps1

This exercises the driver's real helper functions in a temporary fake project, including concurrent unrelated writes, pre-existing assets, post-save edits, Content boundaries, invalid and consumed save manifests, and Content folders that cannot be listed.

The Custom Pass render tests need a rendering device but no window: run them as a gate of their own, without -nullrhi, with -RenderOffscreen -d3d12:

& "<EngineDir>\Engine\Binaries\Win64\UnrealEditor-Cmd.exe" --% "<ProjectDir>\<Project>.uproject" -ExecCmds="Automation RunTests DreamShader.Pass.Render+DreamShader.Pass.Lifecycle; Quit" -RenderOffscreen -d3d12 -unattended -nopause -nosplash -NoDreamShaderEditorBridge -log

Tools/Tests/Invoke-DreamShaderTests.ps1 runs both gates on the standalone host project (Tools/TestHost): the Suite preset under -nullrhi, the Rhi preset offscreen.

Both sides of the Custom Pass gate

Below UE 5.8, DREAMSHADER_WITH_CUSTOM_PASS is 0 and the plugin builds without the renderer half of Custom Pass. A 5.8 machine builds that side too when UBT is given -ProjectDefine:DREAMSHADER_FORCE_NO_CUSTOM_PASS, which the test script's -WithoutCustomPass passes:

pwsh -NoProfile -File Tools\Tests\Invoke-DreamShaderTests.ps1 -Preset Suite -WithoutCustomPass -Filter DreamShader.Lang2.Pipeline+DreamShader.Compiler2.Pipeline+DreamShader.Pass.Logic

At 0 the pipeline tests that build assets are compiled out and DreamShader.Compiler2.Pipeline.NeedsCustomPass is compiled in: a .dsp build reports DSH8300 and writes nothing. The render and lifecycle tests are compiled out with the renderer.

The next build without the switch puts the renderer back by itself: a changed UBT command line invalidates UBT's makefile, so the rules are evaluated again. An environment variable read by a rules file would not be seen — UBT keeps the rules it evaluated last until a rules file changes, and on 5.8 it loads its makefile even under -NoUBTMakefiles. Before running, the script reads which side the binaries are on from the definitions UBT wrote for DreamShaderPass, and stops when it is not the one asked for, as after -NoBuild following a build of the other kind.

Command-line switches

[!WARNING] A skipped test reports success. Every skip path in the suite calls AddInfo(...) and then return true. A -nullrhi run therefore shows DreamShader.Render.ThinCustomVsGraphParity and DreamShader.Roundtrip.MTestToonRenderParity green without having compared a single pixel, and a run on UE 5.3 shows DreamShader.Compiler.Generate.SubstrateMaterial green without having compiled a Substrate material. To prove render parity, run without -nullrhi and read the info lines in the log.

Skip conditions

Every condition under which a test returns success without asserting anything.

Runtime substitutions are shown as {Placeholder} in every message table on this page.

Shared harness

The editor-level tests all use the same scaffolding.

[!NOTE] The Compile/ corpus goldens encode Graph-backend semantics, so that runner pins DefaultBackend = Graph for the duration of each case. Legacy/Compile/ is the exception: its goldens were seeded from the 1.x captures of 2026-09-15 and pin ThinCustom, the project default.

Test index

Every declaration, by translation unit.

DreamShaderAutomationTests.cpp — 30 declarations

Unless the notes say otherwise, each of these writes a source file into DShader/Tests/Automation, generates from it, asserts against the generated asset, and deletes both on the way out.

DreamShaderParameterTests.cpp — 3 declarations

DreamShaderPureFunctionTests.cpp — 8 declarations

No editor, world or asset dependency; these run in milliseconds.

Corpus runners — 13 declarations

The four Compiler2 runners are IMPLEMENT_CUSTOM_COMPLEX_AUTOMATION_TEST with a quiet base (graph auto-layout trips a benign engine ensure); the rest are IMPLEMENT_COMPLEX_AUTOMATION_TEST.

Graph baseline (since 1.9.0)

The automation suite says a compile still succeeds. It does not say the compiler still produces the same graph — and for a compiler rewrite that is the only question. The dump-graph commandlet verb closes that gap: it writes one canonical JSON per generated asset (node classes, reflected properties, connections, pin order; no coordinates, colours or GUIDs), and two captures are compared with an ordinary text diff.

Capture a baseline for a whole project. The compile step is not optional — dump-graph never writes an asset, so an asset already on disk is dumped as it stands rather than rebuilt:

& $UnrealEditorCmd $Project -run=DreamShader compile -All -Force -unattended -nopause -nullrhi -nosplash -stdout -log
& $UnrealEditorCmd $Project -run=DreamShader dump-graph -All -Out="I:/Baseline/before" -unattended -nopause -nullrhi -nosplash -stdout -log

Then, after the change:

git diff --no-index -- I:/Baseline/before I:/Baseline/after

Every hunk is a behavioural difference. ./dsc.ps1 dump-graph -All -Out … is the shorter spelling of the same two commands.

Three automation tests guard the dump itself, and the first is the one that could not be replaced by reading the code — it generates one source twice and asserts the two JSON files are byte-identical, which is the only way to see a leaked GUID, a path-name-seeded node colour, or an object-name suffix:

All three pin the project's DefaultBackend to Graph with FScopedDreamShaderGraphBackendPin, and clean up their source file, their asset and their scratch output tree.

Render parity

Both pixel tests render two materials and compare them frame to frame. ThinCustomVsGraphParity builds twins from the same body, one per backend (Graph and ThinCustom); MTestToonRenderParity compares a project material against the material regenerated from its own decompiled source (Original versus RoundTripTwin).

Parity cases — ThinCustom and Graph twins of the same body, differing only in Backend:

The corpus

Tests/Corpus lives under the plugin, not under the project's DShader tree, so it travels with the plugin and is never picked up by ordinary source discovery. Tests/Corpus/README.md is the reference for the tree, every golden schema and every runner; this section is the map.

Discovery is recursive over the layer's extensions, then sorted.

[!WARNING] Tests/ is not in the release archive. A plugin installed from a release zip has no corpus, so the thirteen runners enumerate zero sub-tests and the suite silently shrinks to its simple declarations. Run the corpus from a repository checkout.

Naming convention

Resulting full test names:

DreamShader.Lang.Parse.TopLevel.T_MinimalShader
DreamShader.Lang2.CorpusIR.Reflected.Rf_LateBoundPin
DreamShader.Compiler2.CorpusLegacy.Material.M_Surface

Entry point per layer

The compile layers copy each fixture into a scratch source root and build into /Game/DreamShaderTests/..., and clean both up.

.expected.json

Every field is opt-in; an absent field asserts nothing. The table below is the Parse/ layer's golden ("entryPoint": "parse"): since the 1.x parser retired in 2.0, its definition fields are read out of the legacy front end's AST and FLegacyMigrationInfo instead of an FTextShaderDefinition, under the same names. Every other layer has its own shape — lang, ir, legacy, legacy-ir, decompile, migrate, compile, roundtrip — documented in Tests/Corpus/README.md. Two conventions they share: errorContains / warningsContain / infosContain hold DSHnnnn codes rather than message text, and a ...Pending: true flag (textPending, irPending, graphPending) skips the byte comparison of a golden nobody has reviewed yet — removing the flag is the reviewer's act, never the writer's.

[!WARNING] outcome is compared only against "error" (case-insensitively, so "Error" and "ERROR" are fine). A typo — "fail", "error " with a trailing space, "errors" — makes the golden expect success, and a fixture that was meant to pin a failure silently starts asserting the opposite. Use "ok" or "error" and nothing else.

Adding a fixture

1. Drop a source into the layer's directory — Tests/Corpus/IR/<Area>/X.dss, Tests/Corpus/Legacy/Compile/Material/M_X.dsm, … Use the layer prefix; add .bad. to the stem for a negative case.
2. Optionally add <same stem>.expected.json with any subset of the fields above. A positive case with no golden still asserts "must succeed"; a .bad. case with no golden asserts "must fail".
3. Write no C++ and do not recompile. Every runner enumerates its tree on the next run.
4. Run the layer once to see the new sub-test appear: -ExecCmds="Automation RunTests DreamShader.Lang.Parse; Quit".
5. To bootstrap a golden from the actual result, add -DreamShaderUpdateGolden and run again, then review the written JSON by hand before committing. A new text, IR or graph golden is written with its ...Pending flag set; delete the flag once you have read the golden line by line.

Regenerating goldens

With -DreamShaderUpdateGolden on the command line, each runner writes the golden instead of asserting it, logging Updated golden '{Path}'. or, on a write failure, Failed to write golden '{Path}'.

What the Parse/ writer emits: entryPoint: "parse", outcome, then either errorContains: [<the codes>] or a definition object with name (omitted when empty), outputDeclarations, outputs, materialFunctions, materialFunction0Kind (only when at least one exists), virtualFunctions, codeNotEmpty and settings (only when non-empty); plus a top-level warningsContain when the parse produced warnings. The other layers' writers are described with their schemas in the corpus README.

Output is pretty-printed JSON.

[!WARNING] -DreamShaderUpdateGolden rewrites goldens from whatever the code currently does. Running it after an unreviewed change accepts a regression as the new baseline. Diff every touched .expected.json before committing, and never put the switch in a CI job.

Diagnostics

Assertion and infrastructure messages the runners emit. Runtime substitutions are shown as {Placeholder}; {Case} is the fixture's absolute source path, not its sub-test name.

The 1.x Generate runner and its generation should … messages are gone (since 2.0.0): its fixtures are Legacy/Compile/, compiled through the 2.0 pipeline. The other layers' assertion messages are in their runners and in Tests/Corpus/README.md.

Example

The shipped negative fixture that pins the "one Shader per parse unit" rule. Tests/Corpus/Parse/TopLevel/T_TwoShaders.bad.dsm:

Shader(Name="DreamShaderTests/Corpus/M_First")
{
    Settings = { Domain = "UI"; ShadingModel = "Unlit"; }
    Outputs = { vec3 Color; Base.EmissiveColor = Color; }
    Graph = { Color = vec3(1.0, 0.0, 0.0); }
}

Shader(Name="DreamShaderTests/Corpus/M_Second")
{
    Settings = { Domain = "UI"; ShadingModel = "Unlit"; }
    Outputs = { vec3 Color; Base.EmissiveColor = Color; }
    Graph = { Color = vec3(0.0, 1.0, 0.0); }
}

The .bad. stem alone would assert "the parse must fail". Its golden, T_TwoShaders.bad.expected.json, also pins the code the legacy front end raises for a second Shader block, DSH2250:

{
	"entryPoint": "parse",
	"outcome": "error",
	"errorContains": ["DSH2250"]
}

Run it:

& "C:\Program Files\Epic Games\UE_5.7\Engine\Binaries\Win64\UnrealEditor-Cmd.exe" --% "D:\Work\MyProject\MyProject.uproject" -ExecCmds="Automation RunTests DreamShader.Lang.Parse.TopLevel; Quit" -nullrhi -unattended -nopause -nosplash -log

The new sub-test appears as:

DreamShader.Lang.Parse.TopLevel.T_TwoShaders.bad

See also

• Contributing — building the plugin and the source-tree layout
• Release — why Tests/ is absent from the release archive
• Commandlet — -run=DreamShader, the other headless entry point
• Editor bridge — what -NoDreamShaderEditorBridge turns off
• Preview — the renderer the parity tests drive
• Backend — Graph vs ThinCustom, the axis the parity cases compare
• Project settings — DefaultBackend, which the Compile/ runner pins to Graph
• DreamShaderLang — ParseDreamShaderLang and every other entry point the text layers call
• Diagnostics index — the messages fixtures assert against
• Version compatibility — DREAMSHADER_WITH_SUBSTRATE_BUILTINS and the UE 5.4 skip