# Setup and Best Practices

## First Steps

This is an in-depth guide that walks you through the asset setup and helps you choose the right settings for your project. Make sure to read the [Quick-Start Manual](https://ipgames.gitbook.io/htrace-ao/quick-start-manual) first and confirm that the asset is enabled and working - no errors, the AO effect is visible, and the [Debug Views](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/shared-settings#debug-mode) look sane. Reset to default settings if needed. This guide is text-only and will not always go deep into how every individual parameter works - for that, refer to the [Settings and Properties](https://ipgames.gitbook.io/htrace-ao/settings-and-properties) section, where each property is described and accompanied by visual examples. It also doesn't cover artistic and management controles such as Intensity, Fadeout and Exclusion / Receiving masks.

You will need a scene (or ideally multiple scenes) to test the asset under various conditions. Make sure these are your actual working scenes, or something close to them in terms of complexity. Don't test on a collection of cube primitives, as that may not be representative of real-world performance or visual output.

Test everything in the **Game** view, since Unity does not write per-object motion vectors in the **Scene** view. Also make sure your resolution is set to whatever you expect your average user to run, **1920×1080 native** is usually a good starting point. When in doubt about how a parameter affects the visual output, enable the [Ambient Occlusion](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/shared-settings#ambient-occlusion) debug mode.

It is also recommended to use the [Unity's Profiler](https://docs.unity3d.com/Manual/profiler-introduction.html) (GPU Usage) when measuring performance. Simply looking at the FPS counter may not be sufficient, since what you see there is the cumulative cost of the entire Unity pipeline - HTrace AO is only one part of it.

*P.S. We recommend reading about all three AO modes in the order they are presented in this guide for the best overall understanding of the asset.*

## **SSAO Mode**

SSAO is the fastest of the three modes and requires neither motion vectors nor a normal buffer to run, but it is the most approximate, especially compared to GTAO and RTAO. You'll mainly want to use it in these cases:

* You don't have motion vectors and aren't ready to implement or fix them just to run AO with temporal denoising.
* Performance doesn't allow for the other modes.
* SSAO looks good enough for your purposes, so there's no point going further.

There are only a few [basic parameters](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/ssao-settings) and no complex setup involved, so it's worth giving it a quick try to see if it meets your needs. If it does, you can stop reading here :)

## GTAO Mode

{% stepper %}
{% step %}
The first thing to determine is the [Resolution](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-quality#resolution) you want to run GTAO at, since this has the biggest performance impact on the effect as a whole. If your scenes contain detailed or thin geometry (foliage, fences, architectural trim, and so on) or "sterile" environments with large bright surfaces (e.g. a sci-fi lab), it's strongly recommended to include them in your testing. Such conditions tend to reveal artifacts and sampling noise more readily, and may raise the quality bar for your setup.

Start with **Quarter** resolution, the fastest of the three, and see if it holds up. If not, move up to **Half**, then **Full**. Half resolution is generally a solid middle ground, though in practice it sits much closer to Full than to Quarter in terms of quality, and only struggles with very thin or fine details. Full resolution gives you the best results at the highest cost.
{% endstep %}

{% step %}
Next, adjust the [Slice Count](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-quality#slice-count). This parameter has a large performance impact but also directly controls the noise level and how much the temporal denoiser needs to do - more slices mean more samples per frame, which means less reliance on temporal accumulation.

* **1 slice** is rarely enough for a stable image on its own, unless you're using [Visibility Bitmasks](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-quality#tracing-mode) + strong [Denoising](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-denoising). It also can work if you're relying on DLSS or another external temporal solution on top, but even then, use it with care.
* **2 slices** is sufficient for most cases, but will likely require the denoiser to some extent.
* **3–4 slices** are only worth considering if you're trying to avoid denoising altogether. Otherwise, the returns diminish quickly.
  {% endstep %}

{% step %}
Once you've settled on a **Slice Count** and **Resolution**, adjust the [World Space Radius](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-visuals#world-space-radius) and [Screen Space Radius](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-visuals#screen-space-radius) if the defaults don't provide enough occlusion coverage for your scene.
{% endstep %}

{% step %}
After that, adjust the [Step Count](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-quality#step-count). Steps (samples) are distributed along the tracing radius and are not automatically scaled as the radius grows, so a larger radius will generally need more steps to maintain quality. Profile carefully, **Step Count** carries a significant performance cost, especially when combined with multiple slices.
{% endstep %}

{% step %}
Now, based on the selected radius, decide whether to switch the [Tracing Mode](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-quality#tracing-mode) to **Visibility Bitmasks**. 

There are two GTAO algorithms available: **Horizon Search** is faster, while **Visibility Bitmasks** are significantly more accurate - particularly at larger radii. If your [World Space Radius](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-visuals#world-space-radius) is 2.5 meters or more, switching to Visibility Bitmasks is strongly recommended, and the UI will output a warning to remind you. At smaller radii, Visibility Bitmasks can still be worthwhile, but they start giving diminishing returns as the radius drops below 1 meter. Their main strength is accurate thickness evaluation for thin geometry (e.g. pillars, bars, fences, rails, grilles, wires, chair and table legs, and so on) preventing the over-occlusion that Horizon Search tends to produce on such objects. If your scene contains this kind of geometry, compare the two modes switching between them to get a feel for the difference. Keep in mind that Visibility Bitmasks come with a substantial performance cost.
{% endstep %}

{% step %}
At this point the GTAO algorithm itself is configured, and we can move on to denoising. If you are planning to use TAA, FSR, STP, DLSS, or any other temporal anti-aliasing or upscaling solution, and especially if you selected **Full** resolution, you can try setting [Denoising Mode](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-denoising#denoising-mode) to **None** to check whether denoising can be offloaded to those algorithms. Note that even if you selected **Half** or **Quarter** resolution, HTrace will force it to **Full** when denoising is set to **None**, and will output a warning, otherwise offloading denoising to an external algorithm would not be feasible. In practice, only the latest versions of DLSS have proven capable of handling this reliably, and even then, disoccluded areas can still exhibit excessive noise. Test this thoroughly before committing to it. If your project has moving or animated characters, objects, or a moving camera, now is the time to bring those into the testing process. Setting [Denoising Mode](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-denoising#denoising-mode) to **None** can yield a significant performance gain by eliminating the denoising pipeline entirely, but this is strictly an experimental approach and should not be considered a general-purpose denoising solution.
{% endstep %}

{% step %}
**Spatial-Only** [Denoising Mode](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-denoising#denoising-mode) should be selected only if temporal denoising is not an option (for example, when valid motion vectors are unavailable) and only when **Slice Count** is set to a sufficiently high value. Try both spatial [Filter Types](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-denoising/spatial-denoising#filter-type) to see which works better for your case. The general rule is to use as little spatial denoising as possible to better preserve occlusion detail. In practice, that means finding the lowest [Pass Count](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-denoising/spatial-denoising#pass-count-box) for the **Box** filter, or the lowest [Filter Radius](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-denoising/spatial-denoising#filter-radius-disk) for the **Disk** filter, that is effective enough for noise suppression and not pushing it further. The **Disk** filter handles fine details better (especially with the [Adaptivity](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-denoising/spatial-denoising#filter-adaptivity-disk) parameter), but tends to leave a slightly grainy look. This is usually unnoticeable on rough surfaces with varied albedo, but can be distracting on cleaner, more uniform materials - in which case the **Box** filter may be the better choice, as it produces smoother results.
{% endstep %}

{% step %}
**Temporal-Only** is a viable option for most cases. It can accumulate enough samples over time, keeps the occlusion as sharp as possible, and is less performance-heavy than the combined **Spatio-Temporal** mode. However, it can struggle in disocclusion areas and wherever occlusion is cast by moving objects (such as a running character) particularly at larger radii. It's worth trying before moving on.

{% hint style="warning" %}
One important thing to keep in mind: even if you want highly reactive AO, temporal denoising is still worth considering for the frame-to-frame smoothing it provides (not to be confused with blur). Without it, as objects enter or leave the frame their occlusion appears and disappears instantaneously, which can read as flickering or pop-in. The same applies when rendering AO at lower resolutions: camera movement can cause a visible snapping or jittering effect on thin geometry. Temporal denoising mitigates both of these issues by smoothing transitions between frames, therefore, sometimes a very fast temporal response is preferable to a strictly instant one.
{% endhint %}

To combat ghosting and occlusion lagging behind moving objects, use the [Rejection Threshold](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-denoising/temporal-denoising#rejection-threshold) and the two available rejection modes. **Simple Rejection** should only be selected when you are severely performance-constrained. Otherwise, leave it disabled, this lets HTrace run more accurate, adaptive (per-pixel) rejection only where it's actually needed, taking into account object velocity, trajectory, and other factors. 

If the occlusion appears too blurry during camera movement, enable [Bicubic](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-denoising/temporal-denoising#bicubic) reprojection. This has a moderate performance cost, so only enable it if it brings a meaningful improvement to the final image.
{% endstep %}

{% step %}
**Spatio-Temporal** is the industry-standard approach and the default denoising mode. Everything covered in the **Spatial-Only** and **Temporal-Only** sections applies here as well, since this mode is a combination of the two. 

It also has one additional trick: if you set the spatial [Filter Type](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/gtao-settings/gtao-denoising/spatial-denoising#filter-type) to **Disk** and reduce the **Filter Radius** to **0**, the spatial pass will only operate on disocclusion areas, where the temporal denoiser had insufficient history, leaving everything else sharp and untouched.
{% endstep %}
{% endstepper %}

## RTAO Mode

{% stepper %}
{% step %}
The first thing to establish with RTAO is whether it is suitable for your project's content and performance targets, as it is the most demanding of the three modes. Ray Tracing does not interact with the same meshes seen (rasterized) by your main camera. Instead, it requires a special structure called RTAS (Ray Tracing Acceleration Structure), which stores scene geometry in a format that rays can intersect against. Depending on the pipeline and the selected [Tracing Mode](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-quality#alpha-cutout-hdrp-unity-2023.2), this structure is either built by Unity or by HTrace, though in both cases the actual construction is handled by Unity's native builder. You need to inspect the RTAS to understand what geometry is present in it. To do so, use one of the following:

* If you are using URP, or HDRP with [Tracing Mode](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-quality#alpha-cutout-hdrp-unity-2023.2) set to **Inline Ray Tracing**, use HTrace's [Hardware Ray Tracing Scene](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/shared-settings#hardware-ray-tracing-scene) visualization debug view.
* If you are using HDRP with [Tracing Mode](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-quality#alpha-cutout-hdrp-unity-2023.2) set to **Native Ray Tracing**, use Unity's Rendering Debugger → Lighting → Fullscreen Debug Mode → RayTracingAccelerationStructure, then set Ray Tracing Acceleration Structure View to **Ambient Occlusion**.

The HTrace debug view has an additional option to highlight mismatches between the RTAS and the rasterized scene in red. A red mesh, or a red portion of one, means it is either absent from the RTAS entirely or appears differently within it. There are many possible causes: GPU Instancing (not automatically included in the RTAS), meshes rendered manually via `Graphics.RenderX`  or similar methods, custom shaders, vertex animation, and so on.

{% hint style="warning" %}
The one expected exception is **alpha-cutout** geometry. Cutout regions will also be highlighted in red, but this is by design, alpha cutout is not evaluated during Inline Ray Tracing, so no action is needed on your end.
{% endhint %}

If the majority of your geometry is missing from the RTAS, or you have a large number of mismatches with the rasterized scene, RTAO will not be able to work to its full potential. In that case, you either need to fix the missing meshes (the exact approach depends on the cause) or switch to GTAO instead. That said, a small amount of missing geometry is perfectly fine: the [Screen Space Pretrace](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-quality#screen-space-pretrace) option exists specifically for this situation and can capture missing occluders in screen space. It works best for small to medium-scale objects, though, so avoid relying on it for large occluders.
{% endstep %}

{% step %}
If the RTAS looks good and your scene is suitable for ray tracing, the next step is to measure RTAO's performance impact and tune the parameters that affect it most. One of such parameters is [Resolution](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-quality#resolution) which works the same way here as in GTAO: **Quarter**, **Half**, and **Full** resolution modes behave identically and carry the same performance implications as described in the [GTAO section](#gtao-mode). 

RTAO also adds a fourth option: **Adaptive**. It typically delivers performance similar to Half resolution, but works differently. Rather than simply rendering every other pixel, it analyzes each frame and applies resolution selectively: areas of high geometric complexity (e.g. foliage, small and thin objects, etc.) are rendered at Half resolution, areas of low complexity at Quarter, and areas lacking sufficient temporal history at Full. It is essentially a dynamic mix of all three modes, applied in a [VRS](https://discussions.unity.com/t/optimize-renderer-features-with-variable-rate-shading-in-unity-6-1/1605893)-like fashion. As with GTAO, we recommend profiling each mode and evaluating it visually before making a decision.
{% endstep %}

{% step %}
The next most important parameter is [Ray Count](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-quality#ray-count). Just like GTAO's Slice Count, it directly controls noise level and temporal responsiveness, and is the primary driver of overall performance:

* **1 ray** is rarely enough for a stable image. As with GTAO, it may work when combined with DLSS, but is generally not recommended.
* **2 rays** can work if your scene has enough geometric detail and albedo variation (a realistic rocky outdoor environment, for example) to mask residual noise. In clean or minimalistic scenes it will likely fall short.
* **3 rays** are suitable for most cases.
* **4 rays** offer a very good balance between performance and visual quality.
* **5–8 rays** should only be used if you have performance headroom to spare, or need top-tier quality with less aggressive denoising. Beyond that, expect diminishing returns.

If you are using **3 rays or fewer**, an additional [Disocclusion Boost](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-denoising/temporal-denoising#disocclusion-boost) property will appear in the [Temporal Denoising](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-denoising/temporal-denoising) UI section. It forces a fixed higher ray budget (4 rays) specifically for pixels in disoccluded areas, speeding up convergence and reducing excessive noise where temporal history is unavailable.
{% endstep %}

{% step %}
If you cannot find a suitable combination of **Resolution** and **Ray Count** that fits your performance budget, consider switching to GTAO instead - there are no other parameters that can meaningfully reduce RTAO's cost. If you have settled on these parameters, review the [Ray Tracing Culling](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-culling) section to see whether any unnecessary objects can be excluded from the RTAS to speed up traversal. This is especially useful in large scenes with dense geometry.
{% endstep %}

{% step %}
At this point the RTAO algorithm itself is configured, and we can move on to denoising. If you haven't already, read the GTAO denoising section [above](#gtao-mode) - it explains when disabling denoising entirely is worth considering. Unlike GTAO, RTAO does not offer separate spatial and temporal stages: the two always run together as a spatio-temporal pair, since the lower sample counts typical of ray tracing require stronger denoising. Within that, HTrace offers two options:

* **Single Pass** is preferable when you have a sufficiently high Ray Count (more than 4) and are running at Half or Full resolution, and/or your scene has enough surface variation to mask residual noise.
* **Double Pass** is the default and suppresses noise more effectively, but introduces more blur to the final result.
  {% endstep %}

{% step %}
The default [Temporal](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-denoising/temporal-denoising) denoising settings will work for most cases. In particular, we recommend keeping the temporal [Sample Count](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-denoising/temporal-denoising#sample-count) at its maximum value of **16** unless you have a specific reason to lower it, such as a very high Ray Count. If your scene contains moving or animated content, make sure to test it and adjust the [Rejection Static](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-denoising/temporal-denoising#rejection-static) and [Rejection Dynamic](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-denoising/temporal-denoising#rejection-dynamic) sliders as needed. Switching the [Reprojection Filter](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-denoising/temporal-denoising#reprojection-filter) to **Bicubic** is also an option if the image looks too blurry during camera movement, though as noted in the [GTAO section](#gtao-mode), verify that it actually improves the final image - in RTAO's case the benefit may be less noticeable, since a more aggressive spatial filter is already running on top.
{% endstep %}

{% step %}
[Spatial](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-denoising/spatial-denoising) filtering follows the same rule as in [GTAO](#gtao-mode): increase the filter radius only as far as needed to suppress noise, and no further. 

RTAO adds one extra feature here: [Contact Occlusion](https://ipgames.gitbook.io/htrace-ao/settings-and-properties/rtao-settings/rtao-denoising/spatial-denoising#contact-occlusion). This is a separate screen-space AO pass with a fixed small radius and its own independent intensity control that runs after the denoiser. **Contact Occlusion** serves two purposes: adding an extra layer of occlusion for a more grounded, contrasty look, and recovering fine detail that ray tracing didn't capture well or that was lost to spatial filtering. It comes at a non-negligible performance cost, so we recommend using it only if it has a meaningful visual impact on your scene.
{% endstep %}
{% endstepper %}