Caching

DreamShader » Generation » Caching

The source-hash short circuit: a stored fingerprint of the compiled text that lets an unchanged source skip regeneration entirely.

Synopsis

digest text    := for each imported header, in the order the include resolver first read it:
                    "// Begin DreamShader source: <project-relative path>\n" <preprocessed header>
                    "\n// End DreamShader source: <project-relative path>\n\n"
                  then the same block for the file itself
                  (a `.dsi` adds its resolved parent and inherited vector channels,
                   a `.dsp` what it references)
build key      := "DSK7|Plugin=<version>|Engine=<major>.<minor>|" <settings> "Defines=<read defines>|"
                  "\n--\n" <digest text>
build key      ->  CRC32  ->  "%08x"  ->  DreamShader.SourceHash   e.g. "9f2c41ab"
source path    ->  project-relative, forward slashes  ->  DreamShader.SourceFile

What is hashed

The hash covers the text the compile actually reads, after preprocessing — not the bytes of the file on disk. (since 2.0.0) Nothing is inlined into the parse any more: each imported .dsh is preprocessed and parsed on its own, and the build key hashes each header's text in its own // Begin DreamShader source: <path> / // End DreamShader source: <path> block, then the file's, including its import lines. A header that two imports reach is hashed once.

It is a build key, not just a source hash

The stamp fingerprints the source in the context that compiles it (since 1.8.0). Anything that changes what a given source produces is folded in, because leaving it out makes the skip check answer "still current" about an asset that is not:

The defines are folded in precisely: only the names the file actually read, each with the value it had, and an explicit <undef> sentinel for one that was read while nothing defined it. Folding the whole table instead would rebuild the world every time any project touched any switch; omitting the sentinel would be worse than either, because then adding a define would not move the hash and the source would quietly start taking a different branch with its asset never rebuilt.

Two properties of that fold are load-bearing. It is sorted by name, case-sensitively, since TMap iteration order is not stable and an unsorted fold would hash differently run to run — which reads as "every asset is stale, every time". And it is injective: a naive Name=Value; join is not, because a value may legitimately contain = and ; (-Define=A=1;B=2 is a valid define), so {A: "1;B=2"} and {A: "1", B: "2"} would fold to the same string and one of them would silently reuse the other's cached asset.

[!NOTE] Before this, the key hashed the source text alone, so changing the backend left every already generated asset looking current — which is why that one setting had its own forced rebuild sweep bolted on. The sweep no longer forces: each affected asset now fails the skip check on its own, and — just as importantly — one the setting does not affect is still skipped instead of being needlessly rebuilt.

Changing the composition of the key invalidates every existing stamp. That costs one rebuild per asset, once, and is the intended effect.

[!NOTE] Editing a .dsh invalidates every source that imports it, but a header never generates anything by itself: handed to the compiler directly it is DSH8296. With Auto Compile On Save on, saving a header queues every source that imports it, directly or through another header, and those compiles find their keys moved. Otherwise the dependents are rebuilt when they are themselves compiled: on their own save, by the startup sweep or the sweep after a change to a setting the build key covers, the Material Content Browser's Compile button, the commandlet, or a cook.

When a batch contains both a file and one it depends on — a header it imports, the parent of a .dsi, a material a .dsp names — the batch is ordered so that dependencies compile first (since 1.8.0). This is not a nicety: a material that calls a ShaderFunction binds its call node against the live UMaterialFunction asset — the emitter reads the pins off the object, not off the source — so compiling the caller first binds it against the previous version of that function's interface. Inside one file the emitter keeps the same order, building a function another product of the file calls before the caller; two that call each other are DSH8299. Files in an import cycle are left in the order the walk reached them; the compile rejects the cycle itself with DSH4211.

Custom Pass pipelines

(since 2.1.0) A .dsp is keyed like any 2.0 source — its preprocessed text, the defines it read, and every setting and version above — plus what its compile read outside the file, because a pipeline whose text did not change still has to be rebuilt when one of those moves:

That is what lets the bridge queue a .dsp unforced on every .usf save and after every rebuild of a material it names: a change that moves none of these is skipped. The shader formats a slot is pre-checked for are not in the key.

A matching key is not enough to skip a pipeline: the slot registry must also still hold every HLSL pass of it, in the slot the asset records, with a snapshot that passed a pre-check and its files on disk, and every exported buffer must have its render target. A registry somebody deleted, or a merge that lost a slot, is rebuilt by the next compile rather than hidden behind an unchanged key.

A .dss that reads an exported buffer through UE.DreamPassBuffer folds in, per node, the pipeline's object path, the buffer, whether it is exported, its format and the render target's path, so a .dsp that changes the export rebuilds the material.

Where the metadata lives

Two keys are written into the generated asset's package metadata, keyed by the asset object.

Storing the project-relative path is deliberate: a checkout on another machine, or a moved project directory, still recognizes its generated assets as its own — the ownership guard and Make Ephemeral read this key. The hash travels with it: the build key names each file by the same project-relative path, so a checkout at another location skips what is current (2.0.0 – 2.1.0 used the absolute path there, and such a checkout rebuilt every asset once).

Which assets get stamped, and when:

[!NOTE] Until 1.8.0 a memory-only Graph material or function was stamped with the source path only, so that the skip could never fire on it. The cost was that nothing could say whether such an asset was current — the Material Content Browser reported every one as stale forever. Every generated asset now carries its hash in both modes, and the entry points that mean "rebuild regardless" force past the skip instead (see below).

A ThinCustom instance additionally carries the source path and the hash as read-only UPROPERTYs — SourceFilePath and SourceHash, category DreamShader — so they are visible in the details panel without inspecting package metadata. SourceFilePath holds the full normalized source path, not the project-relative form the metadata stores; SourceHash is the same eight-hex-digit value. See UDreamShaderMaterialInstance.

[!NOTE] DreamShader.SourceFile doubles as the ownership marker. Its presence is what tells DreamShader that an asset is its own to overwrite, and what Make Ephemeral filters on. See Regeneration.

When regeneration is skipped

The short circuit fires only when all of the following hold:

A skip is reported as the info DSH8237, and the product's result line reads Skipped {ObjectPath} from {File}; source hash is unchanged (build key {BuildKey}). Before the hash is compared, a build that would write to disk is skipped when another editor owns writing this project's generated assets (DSH8209).

A failed package save clears the in-memory source hash, so compiling the same source again retries generation and saving without -Force. The source owner and output digest remain intact: hand edits made after the failed save still stop regeneration. A failed batch invalidates every participating asset, including packages that may already have reached disk. Pipelines also verify their exported render targets' persisted configuration before skipping, so a partially saved batch can be repaired after restarting the editor.

Per asset kind:

Placing the ThinCustom check before the base is created is what makes a skip cheap: no base material, no ownership check, no graph teardown.

[!NOTE] A material function whose usage does not match — a ShaderLayer block whose asset is still marked Default, for instance — is regenerated even when the hash matches, and the usage is corrected.

Forcing regeneration

There is no way to clear the stored hash from the source language. To force a rebuild without a force-capable entry point, either change the source text (any change, including whitespace), or delete the generated asset.

Notes

• The hash is a CRC32, not a cryptographic digest. It detects edits; it is not a security or integrity mechanism.
• There is no generated .ush include to keep in step (since 2.0.0): a Function's HLSL is written into the Custom node of each call, inside the asset, and an unchanged source skips it with everything else. See Generated HLSL.
• The comparison is per asset. One source file that declares a material and three functions stores the same hash on four assets, and each is skipped independently.
• On UE 5.6 and newer the package metadata is accessed through the engine's value-typed metadata API; earlier engines use the object-typed one. The stored keys and values are identical.
• Nothing writes a generation timestamp. Besides these two keys a build stamps DreamShader.OutputDigest and DreamShader.OutputDigestClasses (Divergence), DreamShader.SourceSpans and DreamShader.DecompileHints on an asset with a graph, and DreamShader.PassPipeline on a pipeline's render target. None of them takes part in the skip.

Diagnostics

Example

// DShader/Common.dsh
Function float Remap01(in float value) { return saturate(value * 0.5 + 0.5); }

// DShader/M_Ramp.dsm
import "Common.dsh";

Shader(Name="Materials/M_Ramp")
{
    Properties { ScalarParameter Input = 0.25; }
    Settings   { Domain = "UI"; ShadingModel = "Unlit"; }
    Outputs    { vec3 Color; Base.EmissiveColor = Color; }
    Graph      { float R = Remap01(Input); Color = vec3(R, R, R); }
}

Observed sequence:

save M_Ramp.dsm      Generated Material /Game/Materials/M_Ramp.M_Ramp from ...M_Ramp.dsm.
save M_Ramp.dsm      Skipped /Game/Materials/M_Ramp.M_Ramp from ...M_Ramp.dsm; source hash is unchanged (build key …).
save Common.dsh      (the header builds nothing; it queues M_Ramp.dsm, whose key moved)
                     Generated Material /Game/Materials/M_Ramp.M_Ramp from ...M_Ramp.dsm.

Metadata on the generated instance:

DreamShader.SourceFile   DShader/M_Ramp.dsm
DreamShader.SourceHash   9f2c41ab

See also

• Generation — where the hash is computed in the pipeline
• import — how an imported header is read
• Preprocessor — the defines the key folds in, and why only the ones that were read
• Regeneration — the ownership guard built on DreamShader.SourceFile
• Divergence — the OTHER fingerprint: what the asset holds, not what the source said
• Ephemeral materials — which assets are stamped in which mode
• UDreamShaderMaterialInstance — SourceFilePath and SourceHash
• Generated HLSL — the Custom-node code, and the 1.x include it replaced
• Commandlet — headless compiles and forcing
• HLSL passes — the snapshots whose inputs a .dsp's key covers
• Project settings — auto-compile and debounce