feat: mapper frontend using ElectricSQL ShapeStream (live updates) (#…

…1760)

* feat: first init of basic svelte app for mapper experience

* build: update hotosm/ui and sveltekit deps

* feat: project route with project data on load

* build: use pg logical replication, add electric sync layer

* refactor: replace adapter-static with dyanamic routes

* build: update frontend deps, add electric + pglite

* feat: add test electric-sql modal and page

* build: migration to update task_history.id --> UUID (event_id)

* refactor(backend): update sqlalchemy models to use task_history uuid

* build: remove DEFAULT value from task_history.action_date

* build: ensure sql migration scripts are ran in order

* build: fix base schema task_history.event_id as primary key

* build: remove foreign keys from task_history for electric support

* build: default two retry for migration container

* refactor: remove foreign key relations from DbTaskHistory model sqlalchemy

* build: add compose service for generating electric migrations

* build: add missed justfile for build commands

* build: add openssl to debug dockerfile for electric migrations

* refactor: refactor task history endpoints to use event_id

* build: add npm script for electric migrations

* refactor: update frontend models to use event_id

* ci: add mapper migrations to ignored prettier

* ci: ignore mapper migrations codespell

* build: replace electric js migrations with latest

* refactor: move ts migrations under svelte lib dir

* feat: load electric in via svelte data, split history/comments live query

* refactor: remove task_history sqlalchemy model relationship

* fix: task history routes

* ci: add pnpm-store and node_modules to ignore files

* ci: manually specify svelte plugin on format script

* build: add svelte-maplibre, unocss, turf

* build: configure unocss

* ci: remove parser override for svelte

* feat: maplibre based mapping app start

* feat: start to add styling and electric-sql usage

* refactor: update task history string to include task id & username

* fix: remove action_text string manipulation for task history

* feat: working first version of reactive map colouring

* build: add pako dependency for zlib encoding

* style: add qr-code icon for buttons

* feat: basic task history page, qrcode page

* feat: add missed lib files from mapper frontend

* feat: add missed lib files from mapper frontend

* feat: highlight tasks for selected task id

* feat: start adding an popup task action button to map

* build: remove electric migrations from repo (generated dynamically)

* build: electric service requires migrations to run first

* build: startup order, ignore migrations from repo

* build: use electric-next simplified app

* build: use latest electric-next frontend deps + drizzle orm

* refactor: replace init-electric with init-db for PGLite

* feat: redesign app to use latest electric-next, restructure routes

* build: add mapper app as docker compose service behind proxy

* build: cleanup schema migrations (bigint removed, dup task fields)

* build: also remove redundant task fields no longer used

* refactor: update project get logic to use raw SQL

* fix: fix task_history insert and read logic/endpoints

* fix: flaky splitting on action_text no longer required

* feat(mapper): add buttons to map, finish, reset a task

* build: fix migration dropping task history foreign keys

* build: archive migrations already pushed through, reset numbering

* build: rename reset migration to apply again

* build: add missing custom_tms_url to base schema

* build: add migration to standardise date fields [field]_at

* build: revert rename of _migrations.date_executed due to catch-22

* build: wait for migrations before electric starts

* build: update migration to change all timestamps to with timezone

* refactor: use raw sql to get project, including current task status

* build: remove task_type_prefix from task model (migration)

* docs: update comment on task-events migration for clarity

* refactor: add WKBElement --> geojson conversion back for proj create

* fix(backend): correctly add locked by uid and username to tasks

* fix: correctly display warning on empty odk_token

* fix: get project details only during create/gen files

* build: move submission photo migration revert to correct dir

* refactor: rename th --> latest_th in sql for clarity

* fix: ignore comments when getting latest task action

* fix: fix task comment post endpoint

* build: update electric versions + +svelte + svelte-maplibre

* docs: update info about mapper frontend in readme

* docs: add info about running mapper frontend in local dev

* fix: correctly loading user id into mapper page

* fix: load task areas on mapper page load, use TaskAction enum

* build: comment out debugging symlink to @hotosm/ui

* refactor: @hotosm/ui dist updated upstream

* build: update @hotosm/ui --> 0.2.0b5

* build: archive migrations unrelated to this branch

* fix(backend): remove references to removed 'xforms' db table

* fix(backend): use raw sql for remainder of project creation

* docs: add comment for logic

.gitignore

@@ -26,8 +26,6 @@ dev-dist/
 downloads/
 eggs/
 .eggs/
-lib/
-lib64/
 parts/
 sdist/
 var/

.pre-commit-config.yaml

@@ -53,6 +53,8 @@ repos:
             "!CHANGELOG.md",
             "!CONTRIBUTING.md",
             "!src/frontend/pnpm-lock.yaml",
+            "!src/mapper/pnpm-lock.yaml",
+            "!src/mapper/src/lib/migrations/**",
             "!src/backend/tests/test_data/**",
           ]
 
@@ -74,10 +76,10 @@ repos:
     rev: 3.2.0
     hooks:
       - id: sqlfluff-lint
-        files: ^src/backend/migrations/(?:.*/)*.*$
+        files: ^src/backend/migrations/.*\.sql$
         types: [sql]
       - id: sqlfluff-fix
-        files: ^src/backend/migrations/(?:.*/)*.*$
+        files: ^src/backend/migrations/.*\.sql$
         types: [sql]
 
   # Lint: Bash scripts

contrib/just/build/Justfile

@@ -16,17 +16,27 @@
 #     along with FMTM.  If not, see <https:#www.gnu.org/licenses/>.
 #
 
-# Start all containers
+# Build backend & frontend containers
 [no-cd]
-default:
-  docker compose build
+all: backend frontend
 
-# Build backend API only
+# Build backend container
 [no-cd]
-frontend:
+backend:
   docker compose build api
 
-# Build frontend UI only
+# Build frontend container
 [no-cd]
-backend:
+frontend:
   docker compose build ui
+
+# Set 'dc' alias for docker compose
+alias:
+  #!/usr/bin/env sh
+  dc_alias_cmd="alias dc='docker compose'"
+  if ! grep -q "$dc_alias_cmd" ~/.bashrc; then
+      echo "Adding 'dc' alias to ~/.bashrc."
+      echo "$dc_alias_cmd" | tee -a ~/.bashrc > /dev/null
+  else
+      echo "'dc' alias already exists in ~/.bashrc."
+  fi

docker-compose.development.yml

@@ -242,7 +242,7 @@ services:
     networks:
       - fmtm-net
     entrypoint: ["/migrate-entrypoint.sh"]
-    restart: "on-failure:3"
+    restart: "on-failure:2"
 
   certbot:
     image: "ghcr.io/hotosm/fmtm/proxy:certs-init-development"

docker-compose.main.yml

@@ -162,7 +162,7 @@ services:
     networks:
       - fmtm-net
     entrypoint: ["/migrate-entrypoint.sh"]
-    restart: "on-failure:3"
+    restart: "on-failure:2"
 
   backups:
     image: "ghcr.io/hotosm/fmtm/backend:main"
@@ -174,7 +174,7 @@ services:
     networks:
       - fmtm-net
     entrypoint: ["/backup-entrypoint.sh"]
-    restart: "on-failure:3"
+    restart: "on-failure:2"
     healthcheck:
       test: pg_isready -U ${FMTM_DB_USER} -d ${FMTM_DB_NAME}
       start_period: 5s

docker-compose.staging.yml

@@ -94,7 +94,7 @@ services:
     networks:
       - fmtm-net
     entrypoint: ["/backup-entrypoint.sh"]
-    restart: "on-failure:3"
+    restart: "on-failure:2"
     healthcheck:
       test: pg_isready -U ${FMTM_DB_USER} -d ${FMTM_DB_NAME}
       start_period: 5s

docker-compose.yml

@@ -118,7 +118,6 @@ services:
     volumes:
       - ./src/frontend:/app
       - /app/node_modules/
-      # - ../ui:/app/node_modules/@hotosm/ui:ro
     environment:
       - VITE_API_URL=${API_URL:-http://api.${FMTM_DOMAIN}:${FMTM_DEV_PORT:-7050}}
     ports:
@@ -127,6 +126,25 @@ services:
       - fmtm-net
     restart: "unless-stopped"
 
+  ui-mapper:
+    image: "ghcr.io/hotosm/fmtm/frontend:mapper"
+    build:
+      context: src/mapper
+      dockerfile: debug.dockerfile
+    depends_on:
+      api:
+        condition: service_started
+    volumes:
+      - ./src/mapper:/app
+      - /app/node_modules/
+      - /app/.svelte-kit/
+      # - ../ui:/app/node_modules/@hotosm/ui:ro
+    environment:
+      - VITE_API_URL=${API_URL:-http://api.${FMTM_DOMAIN}:${FMTM_DEV_PORT:-7050}}
+    networks:
+      - fmtm-net
+    restart: "unless-stopped"
+
   central:
     profiles: ["", "central"]
     image: "ghcr.io/hotosm/fmtm/odkcentral:${ODK_CENTRAL_TAG:-v2024.1.0}"
@@ -220,7 +238,7 @@ services:
     # Temp workaround until https://github.com/postgis/docker-postgis/issues/216
     build:
       context: https://github.com/postgis/docker-postgis.git#master:14-3.4/alpine
-    command: -c 'max_connections=300'
+    command: -c 'max_connections=300' -c 'wal_level=logical'
     volumes:
       - fmtm_db_data:/var/lib/postgresql/data/
     environment:
@@ -239,6 +257,28 @@ services:
       timeout: 5s
       retries: 3
 
+  electric:
+    image: "electricsql/electric:${ELECTRIC_TAG:-0.3.4}"
+    depends_on:
+      fmtm-db:
+        condition: service_healthy
+      migrations:
+        condition: service_completed_successfully
+    environment:
+      DATABASE_URL: postgresql://${FMTM_DB_USER:-fmtm}:${FMTM_DB_PASSWORD:-fmtm}@${FMTM_DB_HOST:-fmtm-db}/${FMTM_DB_NAME:-fmtm}
+      # OTEL_EXPORT: otlp
+      # OTLP_ENDPOINT: https://...
+
+      # ELECTRIC_WRITE_TO_PG_MODE: direct_writes
+      # AUTH_JWT_ALG: HS384
+      # AUTH_JWT_KEY: ${ENCRYPTION_KEY}
+      # AUTH_JWT_AUD: ${FMTM_DOMAIN}
+    ports:
+      - "7055:3000"
+    networks:
+      - fmtm-net
+    restart: "unless-stopped"
+
   central-db:
     profiles: ["", "central"]
     image: "postgis/postgis:${POSTGIS_TAG:-14-3.4-alpine}"
@@ -276,6 +316,6 @@ services:
     networks:
       - fmtm-net
     entrypoint: ["/migrate-entrypoint.sh"]
-    restart: "on-failure:3"
+    restart: "on-failure:2"
     healthcheck:
       test: [] # Set the health check test to an empty value to disable it

nginx/templates/dev/fmtm.conf.template

@@ -23,13 +23,47 @@ upstream frontend {
     server ui:7051;
 }
 
+upstream mapper {
+    # Enable sticky sessions based on an incoming client IP address
+    ip_hash;
+
+    server ui-mapper:3000;
+}
+
 server {
     listen 80 default_server;
     server_name fmtm.localhost;
 
     # Max upload size 1GB
     client_max_body_size 1G;
 
+    location /mapnow {
+        proxy_read_timeout 40s;
+        proxy_connect_timeout 20s;
+
+        # Requests headers
+        proxy_set_header Host                $http_host;
+        proxy_set_header X-Real-IP           $remote_addr;
+        proxy_set_header X-Forwarded-Proto   $scheme;
+        proxy_set_header X-Forwarded-For     $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Host    $http_host;
+        proxy_set_header X-Forwarded-Server  $http_host;
+        proxy_set_header X-Forwarded-Port    $server_port;
+
+        # Disable buffer to temp files, tweak buffer for memory
+        proxy_max_temp_file_size 0;
+        proxy_buffer_size 64k;
+        proxy_buffers 8 64k;
+        proxy_busy_buffers_size 64k;
+
+        # Config to enable websockets (for vite dev server)
+        proxy_http_version 1.1;
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection "upgrade";
+
+        proxy_pass http://mapper;
+    }
+
     location / {
         proxy_read_timeout 40s;
         proxy_connect_timeout 20s;
@@ -49,7 +83,7 @@ server {
         proxy_buffers 8 64k;
         proxy_busy_buffers_size 64k;
 
-        # Config to enable websockets
+        # Config to enable websockets (for vite dev server)
         proxy_http_version 1.1;
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection "upgrade";

src/backend/app/auth/auth_routes.py

@@ -168,10 +168,10 @@ async def get_or_create_user(
                 INSERT INTO users (
                     id, username, profile_img, role, mapping_level,
                     is_email_verified, is_expert, tasks_mapped, tasks_validated,
-                    tasks_invalidated, date_registered, last_validation_date
+                    tasks_invalidated, registered_at
                 ) VALUES (
                     :user_id, :username, :profile_img, :role,
-                    'BEGINNER', FALSE, FALSE, 0, 0, 0, NOW(), NOW()
+                    'BEGINNER', FALSE, FALSE, 0, 0, 0, NOW()
                 )
                 ON CONFLICT (id)
                 DO UPDATE SET

src/backend/app/central/central_routes.py

@@ -91,5 +91,5 @@ async def refresh_appuser_token(
     except Exception as e:
         raise HTTPException(
             status_code=400,
-            detail={f"failed to refresh the appuser token for project{project_id}"},
+            detail={f"failed to refresh the appuser token for project {project_id}"},
         ) from e

src/backend/app/central/central_schemas.py

@@ -79,7 +79,7 @@ class CentralBase(BaseModel):
 class Central(CentralBase):
     """ODK Central return, with extras."""
 
-    geometry_geojson: str
+    pass
 
 
 class CentralOut(CentralBase):