Terrain fixes (#188)

* Add set-setup similar to buffers for resource tables.

Added API for creating a resource table set from shader.

* Delete dead code and tidy up

* Fix a few buffers

* Add missing resource table in cmake.

* Add instanced draw command for when you don't care about the base instance

* Fix image layout transitions with TextureGenerateMipmaps.

* Terrain changes

Resolve pass needs to render all geometry again to properly calculate light, but uses Z equals to early out.
Terrain tiles render with instancing.
Terrain PSOs gets created during setup.

* Move bicubic sampling method to util

* Various terrain fixes

Merged together terrain lighting code so it works the same for lowress fallback and virtual sample.
Normals are calculated in the domain shader instead of relying on base mesh normals.
Fix order of operations, page clear now happens after the prepass and the page status copy to readback.
Removed the screen space resolve shader and passes.
Added a bunch of extra markers to the terrain code so it's clearer what's going on when debugging.
Failing to load biome textures reverts to using the system textures for albedo, normals and materials.

* Refactor terrain biome setup API

* Change testviewer to use Array instead of FixedArray for callbacks

* Update system assets

* Split terrain and vegetation settings to separate FBS interfaces

* Update flatbufferinterface to create directories for path before compilation

* Move TBN boilerplate to util.fxh

* Change terrain prepass buffer to R16G16F

* Add support for automatically detecting height maps in dx tex conversion job

* Sample normal from height in resolve.

Move terrain sampling to function to minimize shader code and repetition.

* WIP: Allow for GPU upload size to grow

New API requires getting the offset and buffer to pass to a future upload, since the buffer in question might be different between subsequent calls to upload.

* Fixup for interlocked

Remove the header reimplementation of interlocked in favor of a common header and per-platform implemnentation.
Add AtomicInt, AtomicInt64 and AtomicPointer as types which disallow directly modifying or reading the value.

* Allow for thread safe upload buffer allocation

Utilize 64 bit atomics to keep the byte offset and buffer id in the same object.
This allows for the allocating thread to always know for which buffer allocation attempts where made, such that the appropriate buffer can be returned.
Remove bufferlock for UploadInternal, we should now treat the Upload buffer as being inherently threadsafe.
Replace use of std::pair with Util::Pair.
Reduce upload heap by half, but keep it big enough to fit at least an 8K BC7 texture.

* Fix gcc interlocked compile issue

* More build fixes

* Grrr

* Please be the last

* Add interlocked for 64 bit ints in gccinterlocked

* Fix mistake moving Decrement and Increment outside of namespace

* Fix upload with alignment

And fix uploads in terrain to use proper alignment.

* Simplify aligned offset calculation for upload buffer

* Minor terrain fixes

* Fix lowres terrain to use the same space when calculating normals

---------

Co-authored-by: Gustav Sterbrant <gustav.sterbrant@ttkgames.com>

code/addons/graphicsfeature/CMakeLists.txt

@@ -1,5 +1,5 @@
 nebula_begin_module(graphicsfeature)
-nebula_flatc(SYSTEM graphicsfeature/graphicsfeatureschema.fbs)
+nebula_flatc(SYSTEM graphicsfeature/graphicsfeatureschema.fbs graphicsfeature/terrainschema.fbs graphicsfeature/vegetationschema.fbs)
 fips_deps(render application dynui staticui nflatbuffer)
 target_precompile_headers(graphicsfeature REUSE_FROM application)
 fips_ide_group(features)

code/addons/graphicsfeature/graphicsfeatureunit.cc

@@ -33,9 +33,13 @@
 
 #include "nflatbuffer/nebula_flat.h"
 #include "flat/graphicsfeature/graphicsfeatureschema.h"
+#include "flat/graphicsfeature/terrainschema.h"
+#include "flat/graphicsfeature/vegetationschema.h"
 #include "nflatbuffer/flatbufferinterface.h"
 #include "components/graphicsfeature.h"
 
+#include "terrain/terraincontext.h"
+
 using namespace Graphics;
 using namespace Visibility;
 using namespace Models;
@@ -90,16 +94,22 @@ GraphicsFeatureUnit::OnActivate()
     this->inputServer = Input::InputServer::Create();
     this->gfxServer->Open();
     this->inputServer->Open();
-
-    const IO::URI graphicsFeaturePath("data:tables/graphicsfeature/graphicsfeature.json");
     Util::String contents;
 
-    GraphicsFeature::TerrainSettingsT terrainSettings;
-    if (IO::IoServer::Instance()->ReadFile(graphicsFeaturePath, contents))
+    const IO::URI terrainSetupPath("data:tables/graphicsfeature/terrain.pter");
+    GraphicsFeature::TerrainSetupT terrainSettings;
+    if (IO::IoServer::Instance()->ReadFile(terrainSetupPath, contents))
     {
-        Flat::FlatbufferInterface::DeserializeFlatbuffer<GraphicsFeature::TerrainSettings>(terrainSettings, (byte*)contents.AsCharPtr());
+        Flat::FlatbufferInterface::DeserializeFlatbuffer<GraphicsFeature::TerrainSetup>(terrainSettings, (byte*)contents.AsCharPtr());
     }
 
+    const IO::URI vegetationSetupPath("data:tables/graphicsfeature/vegetation.pveg");
+    GraphicsFeature::VegetationSetupT vegetationSettings;
+    if (IO::IoServer::Instance()->ReadFile(vegetationSetupPath, contents))
+    {
+        Flat::FlatbufferInterface::DeserializeFlatbuffer<GraphicsFeature::VegetationSetup>(vegetationSettings, (byte*)contents.AsCharPtr());
+    }
+
     SizeT width = this->GetCmdLineArgs().GetInt("-w", 1280);
     SizeT height = this->GetCmdLineArgs().GetInt("-h", 1024);
 
@@ -133,20 +143,77 @@ GraphicsFeatureUnit::OnActivate()
     ParticleContext::Create();
     Clustering::ClusterContext::Create(0.01f, 1000.0f, this->wnd);
 
-    Terrain::TerrainSetupSettings settings{
-        terrainSettings.min_height, terrainSettings.max_height,                         // Min/max height 
-        terrainSettings.world_size_width, terrainSettings.world_size_height,            // World size in meters
-        terrainSettings.tile_size_width, terrainSettings.tile_size_height,              // Tile size in meters
-        terrainSettings.quads_per_tile_width, terrainSettings.quads_per_tile_height,    // Amount of quads per tile
-    };
-    Terrain::TerrainContext::Create(settings);
-    Terrain::TerrainContext::SetSun(this->globalLight);
-
-    Vegetation::VegetationSetupSettings vegSettings{
-        terrainSettings.min_height, terrainSettings.max_height,      // min/max height 
-        Math::vec2{ terrainSettings.world_size_width, terrainSettings.world_size_height }
-    };
-    Vegetation::VegetationContext::Create(vegSettings);
+    if (terrainSettings.config->use)
+    {
+        Terrain::TerrainSetupSettings settings {
+            terrainSettings.config->min_height,
+            terrainSettings.config->max_height, // Min/max height
+            terrainSettings.config->world_size_width,
+            terrainSettings.config->world_size_height, // World size in meters
+            terrainSettings.config->tile_size_width,
+            terrainSettings.config->tile_size_height, // Tile size in meters
+            terrainSettings.config->quads_per_tile_width,
+            terrainSettings.config->quads_per_tile_height, // Amount of quads per tile
+        };
+        Terrain::TerrainContext::Create(settings);
+        Terrain::TerrainContext::SetSun(this->globalLight);
+
+        this->terrain.entity = Graphics::CreateEntity();
+        Graphics::RegisterEntity<Terrain::TerrainContext>(this->terrain.entity);
+        Terrain::TerrainContext::SetupTerrain(this->terrain.entity, terrainSettings.instance->height, terrainSettings.instance->decision);
+
+        for (IndexT i = 0; i < terrainSettings.biomes.size(); i++)
+        {
+            const std::unique_ptr<TerrainBiomeSettingsT>& settings = terrainSettings.biomes[i];
+
+            Terrain::BiomeParameters biomeParams =
+            {
+                .slopeThreshold = settings->parameters->slope_threshold,
+                .heightThreshold = settings->parameters->height_threshold,
+                .uvScaleFactor = settings->parameters->uv_scale_factor
+            };
+            Terrain::BiomeSettings biomeSettings = Terrain::BiomeSettingsBuilder()
+                .Parameters(biomeParams)
+                .FlatMaterial(Terrain::BiomeMaterialBuilder()
+                    .Albedo(settings->flat_material->albedo)
+                    .Normal(settings->flat_material->normal)
+                    .Material(settings->flat_material->material)
+                    .Finish()
+                )
+                .SlopeMaterial(Terrain::BiomeMaterialBuilder()
+                    .Albedo(settings->slope_material->albedo)
+                    .Normal(settings->slope_material->normal)
+                    .Material(settings->slope_material->material)
+                    .Finish()
+                )
+                .HeightMaterial(Terrain::BiomeMaterialBuilder()
+                    .Albedo(settings->height_material->albedo)
+                    .Normal(settings->height_material->normal)
+                    .Material(settings->height_material->material)
+                    .Finish()
+                )
+                .HeightSlopeMaterial(Terrain::BiomeMaterialBuilder()
+                    .Albedo(settings->height_slope_material->albedo)
+                    .Normal(settings->height_slope_material->normal)
+                    .Material(settings->height_slope_material->material)
+                    .Finish()
+                )
+                .Mask(settings->mask)
+                .Finish();
+
+            Terrain::TerrainBiomeId biome = Terrain::TerrainContext::CreateBiome(biomeSettings);
+            this->terrain.biomes.Append(biome);
+        }
+
+        if (vegetationSettings.use)
+        {
+            Vegetation::VegetationSetupSettings vegSettings {
+                terrainSettings.config->min_height,
+                terrainSettings.config->max_height, // min/max height
+                Math::vec2 {terrainSettings.config->world_size_width, terrainSettings.config->world_size_height}};
+            Vegetation::VegetationContext::Create(vegSettings);
+        }
+    }    
 
     Lighting::LightContext::Create(frameScript);
     Decals::DecalContext::Create();
@@ -173,7 +240,7 @@ GraphicsFeatureUnit::OnActivate()
     // create environment context for the atmosphere effects
     EnvironmentContext::Create(this->globalLight);
 
-    Util::FixedArray<Graphics::ViewIndependentCall> preLogicCalls =
+    Util::Array<Graphics::ViewIndependentCall> preLogicCalls =
     {
         Im3d::Im3dContext::NewFrame,
         Dynui::ImguiContext::NewFrame,
@@ -184,10 +251,9 @@ GraphicsFeatureUnit::OnActivate()
         EnvironmentContext::OnBeforeFrame,
         EnvironmentContext::RenderUI,
         Particles::ParticleContext::UpdateParticles,
-        //Terrain::TerrainContext::RenderUI
     };
 
-    Util::FixedArray<Graphics::ViewDependentCall> preLogicViewCalls =
+    Util::Array<Graphics::ViewDependentCall> preLogicViewCalls =
     {
         Lighting::LightContext::OnPrepareView,
         Particles::ParticleContext::OnPrepareView,
@@ -197,11 +263,9 @@ GraphicsFeatureUnit::OnActivate()
         Decals::DecalContext::UpdateViewDependentResources,
         Fog::VolumetricFogContext::UpdateViewDependentResources,
         Lighting::LightContext::UpdateViewDependentResources,
-
-        //Terrain::TerrainContext::CullPatches
     };
 
-    Util::FixedArray<Graphics::ViewIndependentCall> postLogicCalls =
+    Util::Array<Graphics::ViewIndependentCall> postLogicCalls =
     {
         Clustering::ClusterContext::UpdateResources,
         ObserverContext::RunVisibilityTests,
@@ -215,12 +279,21 @@ GraphicsFeatureUnit::OnActivate()
         ObserverContext::WaitForVisibility
     };
 
-    Util::FixedArray<Graphics::ViewDependentCall> postLogicViewCalls =
+    Util::Array<Graphics::ViewDependentCall> postLogicViewCalls =
     {
-
-        //Terrain::TerrainContext::UpdateLOD,
-        //Vegetation::VegetationContext::UpdateViewResources
     };
+
+    if (terrainSettings.config->use)
+    {
+        preLogicCalls.Append(Terrain::TerrainContext::RenderUI);
+        preLogicViewCalls.Append(Terrain::TerrainContext::CullPatches);
+        postLogicViewCalls.Append(Terrain::TerrainContext::UpdateLOD);
+
+        if (vegetationSettings.use)
+        {
+            postLogicViewCalls.Append(Vegetation::VegetationContext::UpdateViewResources);
+        }
+    }
 
     this->gfxServer->SetupPreLogicCalls(preLogicCalls);
     this->gfxServer->SetupPreLogicViewCalls(preLogicViewCalls);

code/addons/graphicsfeature/graphicsfeatureunit.h

@@ -17,6 +17,8 @@
 #include "managers/cameramanager.h"
 #include "core/cvar.h"
 
+#include "terrain/terraincontext.h"
+
 //------------------------------------------------------------------------------
 namespace GraphicsFeature
 {
@@ -69,6 +71,9 @@ class GraphicsFeatureUnit : public Game::FeatureUnit
     /// set framescript. must be done before OnActivate!
     void SetFrameScript(IO::URI const& uri);
 
+    /// Setup terrain biome, run before OnActivate
+    void SetupTerrainBiome(const Terrain::BiomeSettings& biomeParameters);
+
     using UIRenderFunc = std::function<void()>;
     /// add a custom UI render function
     void AddRenderUICallback(UIRenderFunc func);
@@ -91,6 +96,11 @@ class GraphicsFeatureUnit : public Game::FeatureUnit
     Game::ManagerHandle graphicsManagerHandle;
     Game::ManagerHandle cameraManagerHandle;
 
+    struct TerrainInstance
+    {
+        Graphics::GraphicsEntityId entity;
+        Util::Array<Terrain::TerrainBiomeId> biomes;
+    } terrain;
     ViewHandle defaultViewHandle;
 
     Core::CVar* r_debug;

code/addons/nflatbuffer/flatbufferinterface.cc

@@ -182,7 +182,8 @@ bool FlatbufferInterface::Compile(IO::URI const& source, IO::URI const& targetFo
             // make sure we have a final / for the output folder
             // FIXME could use own writer
             target += "/";
-            result = flatbuffers::GenerateBinary(*parser, target.AsCharPtr(), filename.AsCharPtr());
+            if (IO::IoServer::Instance()->CreateDirectory(target))
+                result = flatbuffers::GenerateBinary(*parser, target.AsCharPtr(), filename.AsCharPtr());
         }
     }
     delete parser;

code/foundation/CMakeLists.txt

@@ -656,7 +656,7 @@ nebula_begin_module(foundation)
             win32criticalsection.cc
             win32criticalsection.h
             win32event.h
-            win32interlocked.h
+            win32interlocked.cc
             win32readwritelock.cc
             win32readwritelock.h
             win32thread.cc
@@ -672,7 +672,7 @@ nebula_begin_module(foundation)
             linuxthread.h
         )
         fips_dir(threading/gcc GROUP "threading/gcc")
-        fips_files(interlocked.h)
+        fips_files(gccinterlocked.cc)
     else()
         fips_dir(threading/posix GROUP "threading/posix")
         fips_files(

code/foundation/core/types.h

@@ -80,6 +80,9 @@ operator"" _GB(const unsigned long long val)
     return val * 1024 * 1024 * 1024;
 }
 
+//------------------------------------------------------------------------------
+/**
+*/
 #define N_BIT(x) (1 << x)
 template <class MASK, class BITS>
 constexpr MASK
@@ -88,6 +91,9 @@ SetBits(const MASK mask, const BITS bit)
     return MASK(uint(mask) | uint(bit));
 }
 
+//------------------------------------------------------------------------------
+/**
+*/
 template <class MASK, class BITS>
 constexpr MASK
 UnsetBits(const MASK mask, const BITS bit)

code/foundation/memory/win32/win32heap.h

@@ -96,9 +96,9 @@ __forceinline void*
 Win32Heap::Alloc(size_t size)
 {
     #if NEBULA_MEMORY_STATS
-    Threading::Interlocked::Increment(&this->allocCount);
+    Threading::Interlocked::Increment((volatile int*)&this->allocCount);
     // __HeapAlloc16 will always add 16 bytes for memory alignment padding
-    Threading::Interlocked::Add(&this->allocSize, int(size + 16));
+    Threading::Interlocked::Add((volatile int*)&this->allocSize, int(size + 16));
     #endif
     void* ptr = Memory::__HeapAlloc16(this->heap, HEAP_GENERATE_EXCEPTIONS, size);
     return ptr;
@@ -113,7 +113,7 @@ Win32Heap::Realloc(void* ptr, size_t size)
     #if NEBULA_MEMORY_STATS
     size_t curSize = Memory::__HeapSize16(this->heap, 0, ptr);
     // __HeapAlloc16 will always add 16 bytes for memory alignment padding
-    Threading::Interlocked::Add(&this->allocSize, int(size - curSize + 16));
+    Threading::Interlocked::Add((volatile int*)&this->allocSize, int(size - curSize + 16));
     #endif
     void* newPtr = Memory::__HeapReAlloc16(this->heap, HEAP_GENERATE_EXCEPTIONS, ptr, size);
     return newPtr;
@@ -128,8 +128,8 @@ Win32Heap::Free(void* ptr)
     n_assert(0 != ptr);
     #if NEBULA_MEMORY_STATS
     size_t size = Memory::__HeapSize16(this->heap, 0, ptr);
-    Threading::Interlocked::Add(&this->allocSize, -int(size));
-    Threading::Interlocked::Decrement(&this->allocCount);
+    Threading::Interlocked::Add((volatile int*)&this->allocSize, -int(size));
+    Threading::Interlocked::Decrement((volatile int*)&this->allocCount);
     #endif
     BOOL success = Memory::__HeapFree16(this->heap, 0, ptr);
     n_assert(0 != success);

code/foundation/memory/win32/win32memory.cc

@@ -45,10 +45,10 @@ Alloc(HeapType heapType, size_t size, size_t align)
         }
     }
     #if NEBULA_MEMORY_STATS
-        Threading::Interlocked::Increment(&TotalAllocCount);
-        Threading::Interlocked::Add(&TotalAllocSize, (long)size + 16);
-        Threading::Interlocked::Increment(&HeapTypeAllocCount[heapType]);
-        Threading::Interlocked::Add(&HeapTypeAllocSize[heapType], (long)size + 16);
+        Threading::Interlocked::Increment((volatile int*)&TotalAllocCount);
+        Threading::Interlocked::Add((volatile int*)&TotalAllocSize, (long)size + 16);
+        Threading::Interlocked::Increment((volatile int*)&HeapTypeAllocCount[heapType]);
+        Threading::Interlocked::Add((volatile int*)&HeapTypeAllocSize[heapType], (long)size + 16);
         if (MemoryLoggingEnabled && (size >= MemoryLoggingThreshold) &&
             ((MemoryLoggingHeapType == InvalidHeapType) || (MemoryLoggingHeapType == heapType)))
         {
@@ -78,8 +78,8 @@ Realloc(HeapType heapType, void* ptr, size_t size)
     }
     #if NEBULA_MEMORY_STATS
         SIZE_T newSize = __HeapSize16(Heaps[heapType], 0, allocPtr);
-        Threading::Interlocked::Add(&TotalAllocSize, int(newSize - oldSize + 16));
-        Threading::Interlocked::Add(&HeapTypeAllocSize[heapType], int(newSize - oldSize + 16));
+        Threading::Interlocked::Add((volatile int*)&TotalAllocSize, int(newSize - oldSize + 16));
+        Threading::Interlocked::Add((volatile int*)&HeapTypeAllocSize[heapType], int(newSize - oldSize + 16));
         if (MemoryLoggingEnabled && (size >= MemoryLoggingThreshold) &&
             ((MemoryLoggingHeapType == InvalidHeapType) || (MemoryLoggingHeapType == heapType)))
         {
@@ -109,10 +109,10 @@ Free(HeapType heapType, void* ptr)
         #endif
         __HeapFree16(Heaps[heapType], 0, ptr);
         #if NEBULA_MEMORY_STATS
-            Threading::Interlocked::Add(&TotalAllocSize, -int(size));
-            Threading::Interlocked::Decrement(&TotalAllocCount);
-            Threading::Interlocked::Add(&HeapTypeAllocSize[heapType], -int(size));
-            Threading::Interlocked::Decrement(&HeapTypeAllocCount[heapType]);
+            Threading::Interlocked::Add((volatile int*)&TotalAllocSize, -int(size));
+            Threading::Interlocked::Decrement((volatile int*)&TotalAllocCount);
+            Threading::Interlocked::Add((volatile int*)&HeapTypeAllocSize[heapType], -int(size));
+            Threading::Interlocked::Decrement((volatile int*)&HeapTypeAllocCount[heapType]);
             if (MemoryLoggingEnabled && (size >= MemoryLoggingThreshold) &&
                 ((MemoryLoggingHeapType == InvalidHeapType) || (MemoryLoggingHeapType == heapType)))
             {