DEM settings refactoring

* DEM settings refactoring

* linting

* fix formula for z_scaling_factor

* remove superfluous print statement

* fix water depth subtraction

* remove duplicate comment

* Restore sigma

* blur data in the right place

* SRTM Instructions.

* Adjust docs to match dem settings

---------

Co-authored-by: Stan Soldatov <iwatkot@gmail.com>

README.md

@@ -536,13 +536,17 @@ You can also apply some advanced settings to the map generation process.<br>
 
 ### DEM Advanced settings
 
-- Multiplier: the height of the map is multiplied by this value. So the DEM map is just a 16-bit grayscale image, which means that the maximum available value there is 65535, while the actual difference between the deepest and the highest point on Earth is about 20 km. Just note that this setting mostly does not matter, because you can always adjust it in the Giants Editor, learn more about the DEM file and the heightScale parameter in [docs](docs/dem.md). To match the in-game heights with SRTM Data provider, the recommended value is 255 (if easy mode is disabled), but depending on the place, you will need to play with both multiplier and the height scale in Giants Editor to find the best values.
+- Adjust terrain to ground level: Enabling this setting (default) will raise or lower the terrain so that it's lowest point is at ground level (taking into account the plateau and water depth settings).
 
-- Blur radius: the radius of the Gaussian blur filter applied to the DEM map. By default, it's set to 21. This filter just makes the DEM map smoother, so the height transitions will be more natural. You can set it to 1 to disable the filter, but it will result in a Minecraft-like map.
+- Multiplier: DEM multiplier can be used to make the terrain more pronounced. By default the DEM file will be exact copy of the real terrain. If you want to make it more steep, you can increase this value. The recommended value of the multiplier is 1.
 
-- Plateau: this value will be added to each pixel of the DEM image, making it "higher". It's useful when you want to add some negative heights on the map, that appear to be in a "low" place. By default, it's set to 0.
+- Minimum height scale: This value is used as the heightScale in your map i3d. It will automatically be set higher if the elevation in your map (plus plateau, ceiling and water depth) is higher than this value.
 
-- Water depth: this value will be subtracted from each pixel of the DEM image, where water resources are located. Pay attention that it's not in meters, instead it in the pixel value of DEM, which is 16 bit image with possible values from 0 to 65535. When this value is set, the same value will be added to the plateau setting to avoid negative heights.
+- Plateau: DEM plateau value (in meters) is used to make the whole map higher or lower. This value will be added to each pixel of the DEM image, making it higher. It can be useful if you're working on a plain area and need to add some negative height (to make rivers, for example).
+
+- Ceiling: DEM ceiling value (in meters) is used to add padding in the DEM above the highest elevation in your map area. It can be useful if you plan to manually add some height to the map by sculpting the terrain in GE.
+
+- Water depth: Water depth value (in meters) will be subtracted from the DEM image, making the water deeper. The pixel value used for this is calculated based on the heightScale value for your map.
 
 ### Background terrain Advanced settings
 
@@ -677,4 +681,3 @@ But also, I want to thank the people who helped me with the project in some way,
 - [kbrandwijk](https://github.com/kbrandwijk) - for providing [awesome tool](https://github.com/Paint-a-Farm/satmap_downloader) to download the satellite images from the Google Maps and giving a permission to modify it and create a Python Package.
 - [Maaslandmods](https://github.com/Maaslandmods) - for the awesome idea to edit the tree schema in UI, images and code snippets on how to do it.
 - [StrauntMaunt](https://gitlab.com/StrauntMaunt) - for developing procedural generation scripts, providing with the required updates for maps4fs and preparing the docs on how to use procedural generation.
-

docs/dem.md

@@ -1,17 +1,24 @@
 ## Digital Elevation Models (DEM)
+
 DEM is used in Farming Simulator maps to define the terrain height.  
 Every hill, valley, and slope is defined by a DEM. While it may sounds complex, it's really just a 2D grid where each cell has a height value.
+
 ### File description
+
 **Image size:** FS25 -> (map height + 1, map width + 1) FS22 -> (map height / 2 + 1, map width / 2 + 1)  
 **Channels:** 1  
-**Data Type:** uint16  (unsigned 16-bit integer)  
+**Data Type:** uint16 (unsigned 16-bit integer)  
 **File Format:** .png  
 **File Path:** FS25 -> `map_directory/data/dem.png` FS22 -> `map_directory/data/map_dem.png`  
 DEM image is a single channel unsigned 16-bit integer image, which means that each pixel can have an integer value between 0 and 2^16 (65535). So, if the image can have values from 0 to 65535, while the highest point on Earth is 8848 meters, how does it work?
+
 ### Height scale
-And this, where the **heightScale** parameter comes in. It's a multiplier that converts the pixel value to it's in-game height.  By default, in Giants maps, it's value set to 255, but if you're working with DEMs, which contains real-world height values, you should make it much higher. The selection of the actual value is up to you, you can play around with it to get the best result.  
+
+And this, where the **heightScale** parameter comes in. It's a value that converts the pixel value to it's in-game height. By default, in Giants maps, it's value set to 255, which means that the maximum possible value in the DEM image (65535) corresponds to a real world height of 255 meters. The selection of the actual value is up to you, you can play around with it to get the best result.  
 To set this value, you need to open the map.i3d file in Giants Editor, select the terrain on the **Scenegraph** tab, choose **Terrain** tab in the **Attributes** window, and set the **heightScale** parameter. After it you usually need to save the file and reload the map (**File** -> **Reload**).
+
 ### Units per pixel
+
 In Farming Simulator 25 the size of the DEM image is usually the same as the map size but with an additional pixel in each dimension. For example, if the map size is 2048x2048, the DEM image size will be 2049x2049. But in Farming Simulator 22 the DEM image size is half of the map size. So, if the map size is 2048x2048, the DEM image size will be 1025x1025.  
 But actually, it can be changed using the **unitsPerPixel** parameter in the map.i3d file. It defines how many in-game units (meters) each pixel of the DEM image represents. So, in the FS25 by default, it's set to 1, which means that each pixel of the DEM image represents 1 meter in the game. But in FS22 it's set to 2, and that's why the DEM image size is half of the map size.  
-To set this value, you need to open the map.i3d file in Giants Editor, select the terrain on the **Scenegraph** tab, choose **Terrain** tab in the **Attributes** window, and set the **unitsPerPixel** parameter. Just a reminder, it should be an integer value. After it you usually need to save the file and reload the map (**File** -> **Reload**).
+To set this value, you need to open the map.i3d file in Giants Editor, select the terrain on the **Scenegraph** tab, choose **Terrain** tab in the **Attributes** window, and set the **unitsPerPixel** parameter. Just a reminder, it should be an integer value. After it you usually need to save the file and reload the map (**File** -> **Reload**).

maps4fs/generator/component/background.py

@@ -414,10 +414,18 @@ def subtraction(self) -> None:
         water_resources_image = cv2.imread(self.water_resources_path, cv2.IMREAD_UNCHANGED)
         dem_image = cv2.imread(self.output_path, cv2.IMREAD_UNCHANGED)
 
+        # fall back to default value for height_scale 255, it is defined as float | None
+        # but it is always set at this point
+        z_scaling_factor: float = (
+            self.map.shared_settings.mesh_z_scaling_factor
+            if self.map.shared_settings.mesh_z_scaling_factor is not None
+            else 257
+        )
+
         dem_image = self.subtract_by_mask(
             dem_image,
             water_resources_image,
-            self.map.dem_settings.water_depth,
+            int(self.map.dem_settings.water_depth * z_scaling_factor),
         )
 
         # Save the modified dem_image back to the output path

maps4fs/generator/dem.py

@@ -1,6 +1,6 @@
 """This module contains DEM class for processing Digital Elevation Model data."""
 
-import os
+import math
 from typing import Any
 
 import cv2
@@ -31,10 +31,6 @@ class DEM(Component):
     def preprocess(self) -> None:
         self._dem_path = self.game.dem_file_path(self.map_directory)
         self.temp_dir = "temp"
-        self.hgt_dir = os.path.join(self.temp_dir, "hgt")
-        self.gz_dir = os.path.join(self.temp_dir, "gz")
-        os.makedirs(self.hgt_dir, exist_ok=True)
-        os.makedirs(self.gz_dir, exist_ok=True)
 
         self.logger.debug("Map size: %s x %s.", self.map_size, self.map_size)
         self.logger.debug(
@@ -116,24 +112,8 @@ def get_output_resolution(self, use_original: bool = False) -> tuple[int, int]:
         )
         return dem_size, dem_size
 
-    def to_ground(self, data: np.ndarray) -> np.ndarray:
-        """Receives the signed 16-bit integer array and converts it to the ground level.
-        If the min value is negative, it will become zero value and the rest of the values
-        will be shifted accordingly.
-        """
-        # For examlem, min value was -50, it will become 0 and for all values we'll +50.
-
-        if data.min() < 0:
-            self.logger.debug("Array contains negative values, will be shifted to the ground.")
-            data = data + abs(data.min())
-
-        self.logger.debug(
-            "Array was shifted to the ground. Min: %s, max: %s.", data.min(), data.max()
-        )
-        return data
-
     def process(self) -> None:
-        """Reads SRTM file, crops it to map size, normalizes and blurs it,
+        """Reads DTM file, crops it to map size, normalizes and blurs it,
         saves to map directory."""
 
         dem_output_resolution = self.output_resolution
@@ -142,7 +122,7 @@ def process(self) -> None:
         try:
             data = self.dtm_provider.get_numpy()
         except Exception as e:  # pylint: disable=W0718
-            self.logger.error("Failed to get DEM data from SRTM: %s.", e)
+            self.logger.error("Failed to get DEM data from DTM provider: %s.", e)
             self._save_empty_dem(dem_output_resolution)
             return
 
@@ -151,7 +131,7 @@ def process(self) -> None:
             self._save_empty_dem(dem_output_resolution)
             return
 
-        if data.dtype not in ["int16", "uint16"]:
+        if data.dtype not in ["int16", "uint16", "float", "float32"]:
             self.logger.error("DTM provider returned incorrect data type: %s.", data.dtype)
             self._save_empty_dem(dem_output_resolution)
             return
@@ -164,93 +144,158 @@ def process(self) -> None:
             data.max(),
         )
 
-        data = self.to_ground(data)
+        # 1. Resize DEM data to the output resolution.
+        resampled_data = self.resize_to_output(data)
 
-        resampled_data = cv2.resize(
-            data, dem_output_resolution, interpolation=cv2.INTER_LINEAR
-        ).astype("uint16")
+        # 2. Apply multiplier (-10 to 120.4 becomes -20 to 240.8)
+        resampled_data = self.apply_multiplier(resampled_data)
 
-        size_of_resampled_data = asizeof.asizeof(resampled_data) / 1024 / 1024
-        self.logger.debug("Size of resampled data: %s MB.", size_of_resampled_data)
+        # 3. Raise terrain, and optionally lower to plateau level+water depth
+        # e.g. -20 to 240.8m becomes 20 to 280.8m
+        resampled_data = self.raise_or_lower(resampled_data)
+
+        # 4. Determine actual height scale value using ceiling
+        # e.g. 255 becomes 280.8+10 = 291
+        height_scale_value = self.determine_height_scale(resampled_data)
+
+        # 5. Normalize DEM data to 16-bit unsigned integer range (0 to 65535)
+        # e.g. multiply by 65535/291, clip and return as uint16
+        resampled_data = self.normalize_data(resampled_data, height_scale_value)
+
+        # 6. Blur DEM data.
+        resampled_data = self.apply_blur(resampled_data)
+
+        cv2.imwrite(self._dem_path, resampled_data)
+        self.logger.debug("DEM data was saved to %s.", self._dem_path)
+
+        if self.rotation:
+            self.rotate_dem()
 
+    def normalize_data(self, data: np.ndarray, height_scale_value: int) -> np.ndarray:
+        """Normalize DEM data to 16-bit unsigned integer range (0 to 65535).
+
+        Arguments:
+            data (np.ndarray): DEM data.
+            height_scale_value (int): Height scale value.
+
+        Returns:
+            np.ndarray: Normalized DEM data.
+        """
+        normalized_data = np.clip((data / height_scale_value) * 65535, 0, 65535).astype(np.uint16)
         self.logger.debug(
-            "Maximum value in resampled data: %s, minimum value: %s. Data type: %s.",
-            resampled_data.max(),
-            resampled_data.min(),
-            resampled_data.dtype,
+            "DEM data was normalized and clipped to 16-bit unsigned integer range. "
+            "Min: %s, max: %s.",
+            normalized_data.min(),
+            normalized_data.max(),
         )
+        return normalized_data
+
+    def determine_height_scale(self, data: np.ndarray) -> int:
+        """Determine height scale value using ceiling.
+
+        Arguments:
+            data (np.ndarray): DEM data.
+
+        Returns:
+            int: Height scale value.
+        """
+        height_scale = self.map.dem_settings.minimum_height_scale
+        adjusted_height_scale = math.ceil(
+            max(height_scale, data.max() + self.map.dem_settings.ceiling)
+        )
+
+        self.map.shared_settings.height_scale_value = adjusted_height_scale  # type: ignore
+        self.map.shared_settings.mesh_z_scaling_factor = 65535 / adjusted_height_scale
+        self.map.shared_settings.height_scale_multiplier = adjusted_height_scale / 255
+        self.map.shared_settings.change_height_scale = True  # type: ignore
+
+        self.logger.debug("Height scale value is %s.", adjusted_height_scale)
+        return adjusted_height_scale
+
+    def raise_or_lower(self, data: np.ndarray) -> np.ndarray:
+        """Raise or lower terrain to the level of plateau+water depth."""
+
+        if not self.map.dem_settings.adjust_terrain_to_ground_level:
+            return data
+
+        desired_ground_level = self.map.dem_settings.plateau + self.map.dem_settings.water_depth
+        current_ground_level = data.min()
 
-        if self.multiplier != 1:
-            resampled_data = resampled_data * self.multiplier
-
-            self.logger.debug(
-                "DEM data was multiplied by %s. Min: %s, max: %s. Data type: %s.",
-                self.multiplier,
-                resampled_data.min(),
-                resampled_data.max(),
-                resampled_data.dtype,
-            )
-
-            size_of_resampled_data = asizeof.asizeof(resampled_data) / 1024 / 1024
-            self.logger.debug("Size of resampled data: %s MB.", size_of_resampled_data)
-
-            # Clip values to 16-bit unsigned integer range.
-            resampled_data = np.clip(resampled_data, 0, 65535)
-            resampled_data = resampled_data.astype("uint16")
-            self.logger.debug(
-                "DEM data was multiplied by %s and clipped to 16-bit unsigned integer range. "
-                "Min: %s, max: %s.",
-                self.multiplier,
-                resampled_data.min(),
-                resampled_data.max(),
-            )
+        data = data + (desired_ground_level - current_ground_level)
 
         self.logger.debug(
-            "DEM data was resampled. Shape: %s, dtype: %s. Min: %s, max: %s.",
-            resampled_data.shape,
-            resampled_data.dtype,
-            resampled_data.min(),
-            resampled_data.max(),
+            "Array was shifted to the ground level %s. Min: %s, max: %s.",
+            desired_ground_level,
+            data.min(),
+            data.max(),
         )
+        return data
 
-        if self.blur_radius > 0:
-            resampled_data = cv2.GaussianBlur(
-                resampled_data, (self.blur_radius, self.blur_radius), sigmaX=40, sigmaY=40
-            )
-            self.logger.debug(
-                "Gaussion blur applied to DEM data with kernel size %s.",
-                self.blur_radius,
-            )
+    def apply_multiplier(self, data: np.ndarray) -> np.ndarray:
+        """Apply multiplier to DEM data.
+
+        Arguments:
+            data (np.ndarray): DEM data.
 
+        Returns:
+            np.ndarray: Multiplied DEM data.
+        """
+        if not self.multiplier != 1:
+            return data
+
+        multiplied_data = data * self.multiplier
         self.logger.debug(
-            "DEM data was blurred. Shape: %s, dtype: %s. Min: %s, max: %s.",
-            resampled_data.shape,
-            resampled_data.dtype,
-            resampled_data.min(),
-            resampled_data.max(),
+            "DEM data was multiplied by %s. Min: %s, max: %s.",
+            self.multiplier,
+            multiplied_data.min(),
+            multiplied_data.max(),
         )
+        return multiplied_data
 
-        if self.map.dem_settings.plateau:
-            # Plateau is a flat area with a constant height.
-            # So we just add this value to each pixel of the DEM.
-            # And also need to ensure that there will be no values with height greater than
-            # it's allowed in 16-bit unsigned integer.
+    def resize_to_output(self, data: np.ndarray) -> np.ndarray:
+        """Resize DEM data to the output resolution.
 
-            resampled_data += self.map.dem_settings.plateau
-            resampled_data = np.clip(resampled_data, 0, 65535)
+        Arguments:
+            data (np.ndarray): DEM data.
 
-            self.logger.debug(
-                "Plateau with height %s was added to DEM data. Min: %s, max: %s.",
-                self.map.dem_settings.plateau,
-                resampled_data.min(),
-                resampled_data.max(),
-            )
+        Returns:
+            np.ndarray: Resized DEM data.
+        """
+        resampled_data = cv2.resize(data, self.output_resolution, interpolation=cv2.INTER_LINEAR)
 
-        cv2.imwrite(self._dem_path, resampled_data)
-        self.logger.debug("DEM data was saved to %s.", self._dem_path)
+        size_of_resampled_data = asizeof.asizeof(resampled_data) / 1024 / 1024
+        self.logger.debug("Size of resampled data: %s MB.", size_of_resampled_data)
 
-        if self.rotation:
-            self.rotate_dem()
+        return resampled_data
+
+    def apply_blur(self, data: np.ndarray) -> np.ndarray:
+        """Apply blur to DEM data.
+
+        Arguments:
+            data (np.ndarray): DEM data.
+
+        Returns:
+            np.ndarray: Blurred DEM data.
+        """
+        if self.blur_radius == 0:
+            return data
+
+        self.logger.debug(
+            "Applying Gaussion blur to DEM data with kernel size %s.",
+            self.blur_radius,
+        )
+
+        blurred_data = cv2.GaussianBlur(
+            data, (self.blur_radius, self.blur_radius), sigmaX=10, sigmaY=10
+        )
+        self.logger.debug(
+            "DEM data was blurred. Shape: %s, dtype: %s. Min: %s, max: %s.",
+            blurred_data.shape,
+            blurred_data.dtype,
+            blurred_data.min(),
+            blurred_data.max(),
+        )
+        return blurred_data
 
     def rotate_dem(self) -> None:
         """Rotate DEM image."""