Skip to content

Commit

Permalink
update to use whosonfirst/go-whosonfirst-spatial v0.9.0 and whosonfir…
Browse files Browse the repository at this point in the history
…st/go-whosonfirst-spatial-www v0.2.0
  • Loading branch information
sfomuseumbot committed May 22, 2024
1 parent d6aafe4 commit 5a251c3
Show file tree
Hide file tree
Showing 37 changed files with 3,014 additions and 751 deletions.
4 changes: 1 addition & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,4 @@ cli:
server:
go run cmd/server/main.go \
-enable-www \
-spatial-database-uri "sqlite://?dsn=$(DSN)" \
-map-provider leaflet \
-leaflet-tile-url "https://tile.openstreetmap.org/{z}/{x}/{y}.png"
-spatial-database-uri "sqlite://?dsn=$(DSN)"
44 changes: 6 additions & 38 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -410,7 +410,7 @@ pr, err := properties.NewPropertiesReader(ctx, "sqlite://?dsn={DSN}")
## server

```
$>> ./bin/server -h
> ./bin/server -h
-authenticator-uri string
A valid sfomuseum/go-http-auth URI. (default "null://")
-cors-allow-credentials
Expand All @@ -433,12 +433,6 @@ $>> ./bin/server -h
Input data is WOF-flavoured GeoJSON. (Pass a value of '0' or 'false' if you need to index non-WOF documents. (default true)
-iterator-uri string
A valid whosonfirst/go-whosonfirst-iterate/v2 URI. Supported schemes are: directory://, featurecollection://, file://, filelist://, geojsonl://, null://, repo://. (default "repo://")
-leaflet-enable-draw
Enable the Leaflet.Draw plugin.
-leaflet-enable-fullscreen
Enable the Leaflet.Fullscreen plugin.
-leaflet-enable-hash
Enable the Leaflet.Hash plugin. (default true)
-leaflet-initial-latitude float
The initial latitude for map views to use. (default 37.616906)
-leaflet-initial-longitude float
Expand All @@ -447,18 +441,10 @@ $>> ./bin/server -h
The initial zoom level for map views to use. (default 14)
-leaflet-max-bounds string
An optional comma-separated bounding box ({MINX},{MINY},{MAXX},{MAXY}) to set the boundary for map views.
-leaflet-tile-url string
A valid Leaflet 'tileLayer' layer URL. Only necessary if -map-provider is "leaflet".
-log-timings
Emit timing metrics to the application's logger
-map-provider string
The name of the map provider to use. Valid options are: leaflet, protomaps, tangram
-nextzen-apikey string
A valid Nextzen API key. Only necessary if -map-provider is "tangram".
-nextzen-style-url string
A valid URL for loading a Tangram.js style bundle. Only necessary if -map-provider is "tangram". (default "/tangram/refill-style.zip")
-nextzen-tile-url string
A valid Nextzen tile URL template for loading map tiles. Only necessary if -map-provider is "tangram". (default "https://tile.nextzen.org/tilezen/vector/v1/512/all/{z}/{x}/{y}.mvt")
-map-provider-uri string
A valid aaronland/go-http-maps/provider URI. (default "leaflet://?leaflet-tile-url=https://tile.openstreetmap.org/{z}/{x}/{y}.png")
-path-api string
The root URL for all API handlers (default "/api")
-path-data string
Expand All @@ -470,37 +456,19 @@ $>> ./bin/server -h
-path-prefix string
Prepend this prefix to all assets (but not HTTP handlers). This is mostly for API Gateway integrations.
-properties-reader-uri string
A valid whosonfirst/go-reader.Reader URI. Available options are: [fs:// null:// repo:// sqlite:// stdin://]
-protomaps-bucket-uri string
The gocloud.dev/blob.Bucket URI where Protomaps tiles are stored. Only necessary if -map-provider is "protomaps" and -protomaps-serve-tiles is true.
-protomaps-caches-size int
The size of the internal Protomaps cache if serving tiles locally. Only necessary if -map-provider is "protomaps" and -protomaps-serve-tiles is true. (default 64)
-protomaps-database string
The name of the Protomaps database to serve tiles from. Only necessary if -map-provider is "protomaps" and -protomaps-serve-tiles is true.
-protomaps-serve-tiles
A boolean flag signaling whether to serve Protomaps tiles locally. Only necessary if -map-provider is "protomaps".
-protomaps-tile-url string
A valid Protomaps .pmtiles URL for loading map tiles. Only necessary if -map-provider is "protomaps". (default "/tiles/")
A valid whosonfirst/go-reader.Reader URI. Available options are: [fs:// null:// repo:// sqlite:// stdin://]. If the value is {spatial-database-uri} then the value of the '-spatial-database-uri' implements the reader.Reader interface and will be used.
-server-uri string
A valid aaronland/go-http-server URI. (default "http://localhost:8080")
-spatial-database-uri string
A valid whosonfirst/go-whosonfirst-spatial/data.SpatialDatabase URI. options are: [sqlite://]
-tilezen-enable-tilepack
Enable to use of Tilezen MBTiles tilepack for tile-serving. Only necessary if -map-provider is "tangram".
-tilezen-tilepack-path string
The path to the Tilezen MBTiles tilepack to use for serving tiles. Only necessary if -map-provider is "tangram" and -tilezen-enable-tilezen is true.
-verbose
Be chatty.
A valid whosonfirst/go-whosonfirst-spatial/data.SpatialDatabase URI. options are: [rtree:// sqlite://] (default "rtree://")
```

For example:

```
$> bin/server \
-enable-www \
-map-provider tangram \
-spatial-database-uri 'sqlite:///?dsn=modernc:///usr/local/data/sfomuseum-data-architecture.db' \
-nextzen-apikey {NEXTZEN_APIKEY}
-spatial-database-uri 'sqlite:///?dsn=modernc:///usr/local/data/sfomuseum-data-architecture.db'
```

A couple things to note:
Expand Down
2 changes: 1 addition & 1 deletion cmd/server/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"context"
"log/slog"
"os"

_ "github.com/whosonfirst/go-whosonfirst-spatial-sqlite"
"github.com/whosonfirst/go-whosonfirst-spatial-www/app/server"
)
Expand Down
55 changes: 27 additions & 28 deletions database.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@ import (
"github.com/whosonfirst/go-ioutil"
"github.com/whosonfirst/go-reader"
"github.com/whosonfirst/go-whosonfirst-spatial"
"github.com/whosonfirst/go-whosonfirst-spatial-sqlite/wkttoorb"
"github.com/whosonfirst/go-whosonfirst-spatial-sqlite/wkttoorb"
"github.com/whosonfirst/go-whosonfirst-spatial/database"
"github.com/whosonfirst/go-whosonfirst-spatial/filter"
"github.com/whosonfirst/go-whosonfirst-spr/v2"
"github.com/whosonfirst/go-whosonfirst-sqlite-features/v2/tables"
sqlite_spr "github.com/whosonfirst/go-whosonfirst-sqlite-spr/v2"
"github.com/whosonfirst/go-whosonfirst-uri"
"github.com/whosonfirst/go-writer/v3"
"github.com/whosonfirst/go-writer/v3"
)

func init() {
Expand Down Expand Up @@ -438,12 +438,12 @@ func (r *SQLiteSpatialDatabase) getIntersectsByCoord(ctx context.Context, coord
func (r *SQLiteSpatialDatabase) getIntersectsByRect(ctx context.Context, rect *orb.Bound, filters ...spatial.Filter) ([]*RTreeSpatialIndex, error) {

/*
t1 := time.Now()
defer func(){
log.Printf("Time to rect, %v\n", time.Since(t1))
}()
t1 := time.Now()
defer func(){
log.Printf("Time to rect, %v\n", time.Since(t1))
}()
*/

conn, err := r.db.Conn(ctx)

if err != nil {
Expand Down Expand Up @@ -519,13 +519,13 @@ func (r *SQLiteSpatialDatabase) getIntersectsByRect(ctx context.Context, rect *o
func (r *SQLiteSpatialDatabase) inflateResultsWithChannels(ctx context.Context, rsp_ch chan spr.StandardPlacesResult, err_ch chan error, possible []*RTreeSpatialIndex, c *orb.Point, filters ...spatial.Filter) {

/*
t1 := time.Now()
t1 := time.Now()
defer func(){
log.Printf("Time to inflate, %v\n", time.Since(t1))
}()
defer func(){
log.Printf("Time to inflate, %v\n", time.Since(t1))
}()
*/

seen := new(sync.Map)

wg := new(sync.WaitGroup)
Expand All @@ -548,7 +548,6 @@ func (r *SQLiteSpatialDatabase) inflateResultsWithChannels(ctx context.Context,
// will be skipped; if not it will be added (to 'seen') once the spatial index has been successfully inflated.
func (r *SQLiteSpatialDatabase) inflateSpatialIndexWithChannels(ctx context.Context, rsp_ch chan spr.StandardPlacesResult, err_ch chan error, seen *sync.Map, sp *RTreeSpatialIndex, c *orb.Point, filters ...spatial.Filter) {


select {
case <-ctx.Done():
return
Expand All @@ -560,18 +559,18 @@ func (r *SQLiteSpatialDatabase) inflateSpatialIndexWithChannels(ctx context.Cont
feature_id := fmt.Sprintf("%s:%s", sp.FeatureId, sp.AltLabel)

/*
t1 := time.Now()
defer func(){
log.Printf("[%s] Time to inflate w/ channel, %v\n", sp_id, time.Since(t1))
}()
t1 := time.Now()
defer func(){
log.Printf("[%s] Time to inflate w/ channel, %v\n", sp_id, time.Since(t1))
}()
*/

// have we already looked up the filters for this ID?
// see notes below

_, ok := seen.Load(feature_id)

if ok {
return
}
Expand All @@ -588,19 +587,19 @@ func (r *SQLiteSpatialDatabase) inflateSpatialIndexWithChannels(ctx context.Cont
// This is the bottleneck. It appears to be this:
// https://github.com/paulmach/orb/issues/132
// maybe... https://github.com/Succo/wktToOrb/ ?

if strings.HasPrefix(sp.geometry, "[[[") {
// Investigate https://github.com/paulmach/orb/tree/master/geojson#performance
err = json.Unmarshal([]byte(sp.geometry), &poly)
} else {

/*
2023/08/21 22:23:01 [102087463#2084:] orb 20.368308ms
2023/08/21 22:23:01 [102087463#2084:] not-orb 4.206974ms
2023/08/21 22:23:01 [102087463#2084:] orb 20.368308ms
2023/08/21 22:23:01 [102087463#2084:] not-orb 4.206974ms
*/

*/

// poly, err = wkt.UnmarshalPolygon(sp.geometry)

o, err := wkttoorb.Scan(sp.geometry)
Expand All @@ -615,9 +614,9 @@ func (r *SQLiteSpatialDatabase) inflateSpatialIndexWithChannels(ctx context.Cont
if err != nil {
return
}

// END OF maybe move all this code in to whosonfirst/go-whosonfirst-sqlite-features/tables/rtree.go

if !planar.PolygonContains(poly, *c) {
return
}
Expand All @@ -632,7 +631,7 @@ func (r *SQLiteSpatialDatabase) inflateSpatialIndexWithChannels(ctx context.Cont
if ok {
return
}

s, err := r.retrieveSPR(ctx, sp.Path())

if err != nil {
Expand Down
2 changes: 1 addition & 1 deletion database_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (

"github.com/whosonfirst/go-whosonfirst-spatial/database"
"github.com/whosonfirst/go-whosonfirst-spatial/filter"
"github.com/whosonfirst/go-whosonfirst-spatial/geo"
"github.com/whosonfirst/go-whosonfirst-spatial/geo"
)

func TestSpatialDatabaseQuery(t *testing.T) {
Expand Down
5 changes: 3 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ require (
github.com/pkg/errors v0.9.1
github.com/whosonfirst/go-ioutil v1.0.2
github.com/whosonfirst/go-reader v1.0.2
github.com/whosonfirst/go-whosonfirst-spatial v0.8.0
github.com/whosonfirst/go-whosonfirst-spatial-www v0.1.0
github.com/whosonfirst/go-whosonfirst-spatial v0.9.0
github.com/whosonfirst/go-whosonfirst-spatial-www v0.2.0
github.com/whosonfirst/go-whosonfirst-spr/v2 v2.3.7
github.com/whosonfirst/go-whosonfirst-sqlite-features/v2 v2.0.3
github.com/whosonfirst/go-whosonfirst-sqlite-spr/v2 v2.0.0
Expand Down Expand Up @@ -64,6 +64,7 @@ require (
github.com/benbjohnson/clock v1.3.0 // indirect
github.com/bits-and-blooms/bitset v1.2.0 // indirect
github.com/cenkalti/backoff/v4 v4.2.1 // indirect
github.com/dhconnelly/rtreego v1.2.0 // indirect
github.com/dominikbraun/graph v0.23.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/fsnotify/fsnotify v1.6.0 // indirect
Expand Down
10 changes: 6 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,8 @@ github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGX
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dhconnelly/rtreego v1.2.0 h1:LWhGPhw+iGuhg8hmHA/H8WV60qKtzecOjii0FMevGlk=
github.com/dhconnelly/rtreego v1.2.0/go.mod h1:SDozu0Fjy17XH1svEXJgdYq8Tah6Zjfa/4Q33Z80+KM=
github.com/dominikbraun/graph v0.23.0 h1:TdZB4pPqCLFxYhdyMFb1TBdFxp8XLcJfTTBQucVPgCo=
github.com/dominikbraun/graph v0.23.0/go.mod h1:yOjYyogZLY1LSG9E33JWZJiq5k83Qy2C6POAuiViluc=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
Expand Down Expand Up @@ -338,10 +340,10 @@ github.com/whosonfirst/go-whosonfirst-reader v1.0.2 h1:yLsDd6GJOsfMVJz63noH5Haoo
github.com/whosonfirst/go-whosonfirst-reader v1.0.2/go.mod h1:1M6osxb9qXB3+ARINW4vuEaCTIN55abLRBGbBtfpGJ0=
github.com/whosonfirst/go-whosonfirst-sources v0.1.0 h1:JuKLa6KWke22jBfJ1pM9WQHoz1/3pbDv2C+aR+THPPQ=
github.com/whosonfirst/go-whosonfirst-sources v0.1.0/go.mod h1:EUMHyGzUmqPPxlMmOp+28BFeoBdxxE0HCKRd67lkqGM=
github.com/whosonfirst/go-whosonfirst-spatial v0.8.0 h1:Luqn35ePrGl9O8F2uWh1+rn1D1JRxgdrqU0Wyvs/wtQ=
github.com/whosonfirst/go-whosonfirst-spatial v0.8.0/go.mod h1:YF4SVGfbcoRE5KgcISF2qd0Bnyvqvb1wNKL8or4iK18=
github.com/whosonfirst/go-whosonfirst-spatial-www v0.1.0 h1:8q/e8e550NMijQil2Yenx9VKQ8o99e5JcRcXEKv+wVc=
github.com/whosonfirst/go-whosonfirst-spatial-www v0.1.0/go.mod h1:nta22frPrzrViLkPMDI6ZZyf3m4M0C3W4dcPCGugCXE=
github.com/whosonfirst/go-whosonfirst-spatial v0.9.0 h1:uDIqg36IKvGF1XIIQnhQX/8w21dcWH3P1WPjKlzt+bk=
github.com/whosonfirst/go-whosonfirst-spatial v0.9.0/go.mod h1:ZZDfswfjQliuqqNCrpLHW2LVwR7S7QaVMtFiTdvEQH0=
github.com/whosonfirst/go-whosonfirst-spatial-www v0.2.0 h1:LpjuvmBznHQnYbYquo0YPYSJy0dab2sBRP5t6I5A9HU=
github.com/whosonfirst/go-whosonfirst-spatial-www v0.2.0/go.mod h1:yoPKlKjXg0fdRnMuZhcuurn2mBKnN3dMli/4w8TYGas=
github.com/whosonfirst/go-whosonfirst-spr-geojson v0.0.8 h1:K0rCaKo0xvOtuS/x9r62bEfIK3OTdZqbDmgG+A3xDSs=
github.com/whosonfirst/go-whosonfirst-spr-geojson v0.0.8/go.mod h1:Nh7PwBMDT1kUrp9jBMFw+Ilue6up1tz2KhE39btS/ZU=
github.com/whosonfirst/go-whosonfirst-spr/v2 v2.3.7 h1:1j4IMuVua4/NhqKm2ke16h2V5Z2XiLoOdXbjDrfZb0E=
Expand Down
1 change: 1 addition & 0 deletions vendor/github.com/dhconnelly/rtreego/.gitignore

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions vendor/github.com/dhconnelly/rtreego/.travis.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

29 changes: 29 additions & 0 deletions vendor/github.com/dhconnelly/rtreego/LICENSE

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 5a251c3

Please sign in to comment.