Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reimplement managed raster class in C++ #1597

Merged
Merged
Changes from 1 commit
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
5e58f60
compare different implementations of upslope neighbors
emlys May 7, 2024
7c51c08
improve efficiency of managedraster helper functions
emlys Jun 15, 2024
a55d57c
work in progress: rewrite managed raster module in c++
emlys Jun 25, 2024
068d886
work in progress: managed raster in C++
emlys Jul 16, 2024
8576088
C++ managed raster implementation now faster on SDR sample data
emlys Jul 17, 2024
7f2ec80
Merge branch 'feature/routing-refactor' into feature/routing-refactor…
emlys Jul 17, 2024
11ef40f
NDR tests passing with new C++ implementation of managedraster
emlys Jul 18, 2024
c7486f2
all tests using managedraster passing locally
emlys Jul 23, 2024
048e769
debugging: find gdal.h
emlys Jul 23, 2024
cccf0e6
debugging: add incldue path to compile args for windows
emlys Jul 23, 2024
8d104a4
debugging
emlys Jul 23, 2024
6ca0f78
debugging'
emlys Jul 23, 2024
61d1f65
debugging
emlys Jul 23, 2024
d3aebac
debugging
emlys Jul 23, 2024
eb15fa1
debugging
emlys Jul 23, 2024
93ec4b9
debugging
emlys Jul 23, 2024
4506414
debugging
emlys Jul 23, 2024
1ff72b6
debugging
emlys Jul 23, 2024
6acaea6
debugging
emlys Jul 23, 2024
9316d20
use C++20; no need to compile managed_raster as a separate extension
emlys Jul 23, 2024
0d4805a
debugging
emlys Jul 23, 2024
53b15d0
debugging
emlys Jul 23, 2024
06c8648
debugging
emlys Jul 23, 2024
1749eb9
debugging
emlys Jul 23, 2024
e4c234f
debugging
emlys Jul 23, 2024
bbd7c5a
debugging
emlys Jul 23, 2024
7bc4b58
debugging
emlys Jul 23, 2024
efc5527
debugging
emlys Jul 23, 2024
474a89a
debugging
emlys Jul 23, 2024
a63532f
debugging
emlys Jul 23, 2024
05aa968
debugging
emlys Jul 23, 2024
81af8a3
debugging
emlys Jul 23, 2024
a0e5810
debugging
emlys Jul 23, 2024
d5bd998
use setuptools library_dirs parameter
emlys Jul 24, 2024
09a6f8e
debugging
emlys Jul 24, 2024
c669180
organizing options in setup.py
emlys Jul 24, 2024
61883f5
swy core: use libcpp.vector rather than allocating an array
emlys Jul 25, 2024
d236af1
swy core: use libcpp.vector rather than allocating an array
emlys Jul 25, 2024
7e12302
use std::string type to store raster path in ManagedRaster
emlys Jul 30, 2024
c664e43
Merge branch 'main' into feature/routing-refactor-compare
emlys Aug 9, 2024
baac039
Merge branch 'main' into feature/routing-refactor-compare
emlys Aug 22, 2024
8cf1e6b
Merge branch 'feature/routing-refactor' into feature/routing-refactor…
emlys Aug 22, 2024
5932fdb
remove extra changed files
emlys Aug 22, 2024
8fe87ec
clean upg
emlys Aug 22, 2024
d7d3b09
now slightly faster than implementation on main
emlys Aug 26, 2024
4e5e717
add library dir for gdal to setup.py
emlys Aug 26, 2024
5f6a647
typo
emlys Aug 26, 2024
8defb6f
another mistake
emlys Aug 26, 2024
6245b6a
add C++ errors for invalid arguments to managedraster
emlys Sep 3, 2024
364d7ff
Update src/natcap/invest/managed_raster/ManagedRaster.h
emlys Oct 15, 2024
386beb2
remove unneeded file
emlys Oct 15, 2024
7b994e2
remove unused enumeration
emlys Oct 15, 2024
f62a696
inline managedraster set and get functions
emlys Oct 15, 2024
8109a60
write C++ error messages out to error stream
emlys Oct 15, 2024
da23f3e
clean up lru_cache and actualBlockWidths in managedraster.close
emlys Oct 23, 2024
31565e8
inline is_close function
emlys Oct 24, 2024
8c31a12
use gdal-config to get library path on macos
emlys Oct 24, 2024
b88567c
separate C++ classes for each variant of neighbor pixel iteration
emlys Oct 24, 2024
36c26b2
drafting a true C++ iterator
emlys Oct 25, 2024
fc8dd38
implement true C++ iterator for downslope neighbors and use in SDR
emlys Oct 29, 2024
e6a370e
using downslope iterators everywhere
emlys Oct 29, 2024
2cf8f5d
use true iterators for upslope neighbors
emlys Oct 29, 2024
ba37a0a
Merge branch 'feature/routing-refactor' into feature/routing-refactor…
emlys Oct 29, 2024
b957881
debugging
emlys Oct 29, 2024
989a7b6
debugging
emlys Oct 29, 2024
b8e710f
cdef flow_dir_sum
emlys Oct 29, 2024
b532001
debugging
emlys Oct 29, 2024
9421e2c
debugging
emlys Oct 29, 2024
9981569
debugging
emlys Oct 29, 2024
6d556e2
debugging
emlys Oct 31, 2024
c7742e8
debugging
emlys Oct 31, 2024
7b9db1f
debugging
emlys Oct 31, 2024
9071362
debugging
emlys Oct 31, 2024
ba9cebe
debugging
emlys Oct 31, 2024
13cc0dc
debugging
emlys Oct 31, 2024
7a767f5
use const reference parameters to equality fns
emlys Oct 31, 2024
5b8c04e
inherit from Neighbors class
emlys Oct 31, 2024
947900c
inherit operator funcs
emlys Nov 1, 2024
f737225
inherit equality functions from parent class
emlys Nov 1, 2024
3a623eb
run all tests
emlys Nov 1, 2024
420145f
inherit increment operators from parent class
emlys Nov 1, 2024
dde78df
call base class constructors where possible
emlys Nov 1, 2024
11cc0ef
setup.py: use same options for linux as for mac
emlys Nov 5, 2024
25c954a
remove extra comments
emlys Nov 22, 2024
d50a2c9
downgrade compiler version to c++17
emlys Dec 9, 2024
3adae2d
read gdal lib path from env variable on windows
emlys Dec 10, 2024
24d1028
set NATCAP_INVEST_GDAL_LIB_PATH in github actions
emlys Dec 10, 2024
b3d2a55
bump c++ version back to 20
emlys Dec 10, 2024
45ec188
set NATCAP_INVEST_GDAL_LIB_PATH in one more place
emlys Dec 10, 2024
9b95796
temporarily pin pytest-subtests version
emlys Dec 10, 2024
3c9b795
Merge branch 'feature/routing-refactor' into feature/routing-refactor…
emlys Dec 10, 2024
07059b0
remove stdlib flag from compile command; default to libstdc++
emlys Dec 10, 2024
041cd1a
remove managed_raster pyx file
emlys Dec 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
separate C++ classes for each variant of neighbor pixel iteration
emlys committed Oct 24, 2024
commit b88567c6997a15a8721396a715cbcac5345332b3
51 changes: 43 additions & 8 deletions src/natcap/invest/managed_raster/ManagedRaster.h
Original file line number Diff line number Diff line change
@@ -433,8 +433,18 @@ class DownslopeNeighborIterator {
return next();
}
}
};

class DownslopeNeighborIteratorNoSkip: public DownslopeNeighborIterator {
public:

DownslopeNeighborIteratorNoSkip() {}

NeighborTuple next_no_skip() {
DownslopeNeighborIteratorNoSkip(ManagedFlowDirRaster managed_raster, int x, int y)
: DownslopeNeighborIterator(managed_raster, x, y) // Call the superclass constructor in the subclass' initialization list.
{ }

NeighborTuple next() {
NeighborTuple n;
long xj, yj;
int flow_ij;
@@ -455,7 +465,7 @@ class DownslopeNeighborIterator {
return n;
} else {
n_dir += 1;
return next_no_skip();
return next();
}
}
};
@@ -518,8 +528,18 @@ class UpslopeNeighborIterator {
return next();
}
}
};

class UpslopeNeighborIteratorNoDivide: public UpslopeNeighborIterator {
public:

UpslopeNeighborIteratorNoDivide() { }

NeighborTuple next_no_divide() {
UpslopeNeighborIteratorNoDivide(ManagedFlowDirRaster managed_raster, int x, int y)
: UpslopeNeighborIterator(managed_raster, x, y)
{}

NeighborTuple next() {

NeighborTuple n;
long xj, yj;
@@ -537,7 +557,7 @@ class UpslopeNeighborIterator {
if (xj < 0 or xj >= raster.raster_x_size or
yj < 0 or yj >= raster.raster_y_size) {
n_dir += 1;
return next_no_divide();
return next();
}

flow_dir_j = raster.get(xj, yj);
@@ -549,11 +569,25 @@ class UpslopeNeighborIterator {
return n;
} else {
n_dir += 1;
return next_no_divide();
return next();
}
}
};

NeighborTuple next_skip(int skip) {

class UpslopeNeighborIteratorSkip: public UpslopeNeighborIterator {
public:
int skip;

UpslopeNeighborIteratorSkip() { }

UpslopeNeighborIteratorSkip(ManagedFlowDirRaster managed_raster, int x, int y, int skip)
: UpslopeNeighborIterator(managed_raster, x, y)
{
this->skip = skip;
}

NeighborTuple next() {

NeighborTuple n;
long xj, yj;
@@ -573,7 +607,7 @@ class UpslopeNeighborIterator {
yj < 0 or yj >= raster.raster_y_size or
INFLOW_OFFSETS[n_dir] == skip) {
n_dir += 1;
return next_skip(skip);
return next();
}

flow_dir_j = raster.get(xj, yj);
@@ -590,11 +624,12 @@ class UpslopeNeighborIterator {
return n;
} else {
n_dir += 1;
return next_skip(skip);
return next();
}
}
};


inline bool is_close(double x, double y) {
if (isnan(x) and isnan(y)) {
return true;
37 changes: 34 additions & 3 deletions src/natcap/invest/managed_raster/managed_raster.pxd
Original file line number Diff line number Diff line change
@@ -87,7 +87,18 @@ cdef extern from "ManagedRaster.h":
DownslopeNeighborIterator()
DownslopeNeighborIterator(ManagedFlowDirRaster, int, int)
NeighborTuple next()
NeighborTuple next_no_skip()

cdef cppclass DownslopeNeighborIteratorNoSkip:
ManagedFlowDirRaster raster
int col
int row
int n_dir
int flow_dir
int flow_dir_sum

DownslopeNeighborIteratorNoSkip()
DownslopeNeighborIteratorNoSkip(ManagedFlowDirRaster, int, int)
NeighborTuple next()

cdef cppclass UpslopeNeighborIterator:
ManagedFlowDirRaster raster
@@ -99,8 +110,28 @@ cdef extern from "ManagedRaster.h":
UpslopeNeighborIterator()
UpslopeNeighborIterator(ManagedFlowDirRaster, int, int)
NeighborTuple next()
NeighborTuple next_no_divide()
NeighborTuple next_skip(int skip)

cdef cppclass UpslopeNeighborIteratorNoDivide:
ManagedFlowDirRaster raster
int col
int row
int n_dir
int flow_dir

UpslopeNeighborIteratorNoDivide()
UpslopeNeighborIteratorNoDivide(ManagedFlowDirRaster, int, int)
NeighborTuple next()

cdef cppclass UpslopeNeighborIteratorSkip:
ManagedFlowDirRaster raster
int col
int row
int n_dir
int flow_dir

UpslopeNeighborIteratorSkip()
UpslopeNeighborIteratorSkip(ManagedFlowDirRaster, int, int, int)
NeighborTuple next()

bint is_close(double, double)

10 changes: 5 additions & 5 deletions src/natcap/invest/ndr/ndr_core.pyx
Original file line number Diff line number Diff line change
@@ -13,7 +13,7 @@ from libcpp.stack cimport stack
from libc.math cimport exp
from ..managed_raster.managed_raster cimport ManagedRaster
from ..managed_raster.managed_raster cimport ManagedFlowDirRaster
from ..managed_raster.managed_raster cimport DownslopeNeighborIterator
from ..managed_raster.managed_raster cimport DownslopeNeighborIteratorNoSkip
from ..managed_raster.managed_raster cimport UpslopeNeighborIterator
from ..managed_raster.managed_raster cimport NeighborTuple
from ..managed_raster.managed_raster cimport is_close
@@ -185,15 +185,15 @@ def ndr_eff_calculation(
else:
working_retention_eff = 0.0

dn_iterator = DownslopeNeighborIterator(
dn_iterator = DownslopeNeighborIteratorNoSkip(
mfd_flow_direction_raster, global_col, global_row)
has_outflow = False
neighbor = dn_iterator.next_no_skip()
neighbor = dn_iterator.next()
while neighbor.direction < 8:
has_outflow = True
if (neighbor.x < 0 or neighbor.x >= n_cols or
neighbor.y < 0 or neighbor.y >= n_rows):
neighbor = dn_iterator.next_no_skip()
neighbor = dn_iterator.next()
continue
if neighbor.direction % 2 == 1:
step_size = <float>(cell_size * 1.41421356237)
@@ -227,7 +227,7 @@ def ndr_eff_calculation(

working_retention_eff += (
intermediate_retention * neighbor.flow_proportion)
neighbor = dn_iterator.next_no_skip()
neighbor = dn_iterator.next()

if has_outflow:
working_retention_eff /= dn_iterator.flow_dir_sum
11 changes: 7 additions & 4 deletions src/natcap/invest/sdr/sdr_core.pyx
Original file line number Diff line number Diff line change
@@ -13,6 +13,7 @@ from ..managed_raster.managed_raster cimport ManagedFlowDirRaster
from ..managed_raster.managed_raster cimport is_close
from ..managed_raster.managed_raster cimport DownslopeNeighborIterator
from ..managed_raster.managed_raster cimport UpslopeNeighborIterator
from ..managed_raster.managed_raster cimport UpslopeNeighborIteratorSkip


cdef extern from "time.h" nogil:
@@ -118,6 +119,8 @@ def calculate_sediment_deposition(
cdef time_t last_log_time = ctime(NULL)
cdef float f_j_weighted_sum
cdef NeighborTuple neighbor
cdef UpslopeNeighborIterator up_iterator
cdef UpslopeNeighborIteratorSkip up_iterator_skip

for offset_dict in pygeoprocessing.iterblocks(
(mfd_flow_direction_path, 1), offset_only=True, largest_block=0):
@@ -205,16 +208,16 @@ def calculate_sediment_deposition(
# completed
upslope_neighbors_processed = 1
# iterate over each neighbor-of-neighbor
up_iterator = UpslopeNeighborIterator(
mfd_flow_direction_raster, neighbor.x, neighbor.y)
neighbor_of_neighbor = up_iterator.next_skip(neighbor.direction)
up_iterator_skip = UpslopeNeighborIteratorSkip(
mfd_flow_direction_raster, neighbor.x, neighbor.y, neighbor.direction)
neighbor_of_neighbor = up_iterator_skip.next()
while neighbor_of_neighbor.direction < 8:
if is_close(sediment_deposition_raster.get(
neighbor_of_neighbor.x, neighbor_of_neighbor.y
), target_nodata):
upslope_neighbors_processed = 0
break
neighbor_of_neighbor = up_iterator.next_skip(neighbor.direction)
neighbor_of_neighbor = up_iterator_skip.next()
# if all upslope neighbors of neighbor j are
# processed, we can push j onto the stack.
if upslope_neighbors_processed:
Original file line number Diff line number Diff line change
@@ -21,6 +21,7 @@ from ..managed_raster.managed_raster cimport ManagedRaster
from ..managed_raster.managed_raster cimport ManagedFlowDirRaster
from ..managed_raster.managed_raster cimport DownslopeNeighborIterator
from ..managed_raster.managed_raster cimport UpslopeNeighborIterator
from ..managed_raster.managed_raster cimport UpslopeNeighborIteratorNoDivide
from ..managed_raster.managed_raster cimport NeighborTuple
from ..managed_raster.managed_raster cimport is_close

@@ -217,8 +218,8 @@ cpdef calculate_local_recharge(
# initialize to 0 so we indicate we haven't tracked any
# mfd values yet
l_sum_avail_i = 0.0
up_iterator = UpslopeNeighborIterator(flow_raster, xi, yi)
neighbor = up_iterator.next_no_divide()
up_iterator = UpslopeNeighborIteratorNoDivide(flow_raster, xi, yi)
neighbor = up_iterator.next()
mfd_dir_sum = 0
while neighbor.direction < 8:
# pixel flows inward, check upslope
@@ -233,7 +234,7 @@ cpdef calculate_local_recharge(
l_sum_avail_i += (
l_sum_avail_j + l_avail_j) * neighbor.flow_proportion
mfd_dir_sum += <int>neighbor.flow_proportion
neighbor = up_iterator.next_no_divide()
neighbor = up_iterator.next()
# calculate l_sum_avail_i by summing all the valid
# directions then normalizing by the sum of the mfd
# direction weights (Equation 8)