Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add census data maps #188

Merged
merged 11 commits into from
Jan 22, 2025
Merged

feat: add census data maps #188

merged 11 commits into from
Jan 22, 2025

Conversation

joaquimds
Copy link

@joaquimds joaquimds commented Jan 20, 2025

Description

  • Commands to import output areas and MSOA/LSOA
  • GraphQL query for choropleth data can be filtered by map bounds
  • Front-end automatically selects the tileset for the zoom level
    (census boundary type only)

Motivation and Context

Required for https://linear.app/commonknowledge/project/hnh-ingest-and-visualise-a-uk-wide-mrp-dataset-d1c00221b2d5/issues

How Can It Be Tested?

  1. Download Output Areas from here: https://www.data.gov.uk/dataset/4d4e021d-fe98-4a0e-88e2-3ead84538537/output-areas-december-2021-boundaries-ew-bgc-v21 and save as data/output_areas.geojson
  2. Import MSOA and LSOA areas with python manage.py import_areas
  3. Download sample census data from here: https://www.nomisweb.co.uk/sources/census_2021_bulk (I used TS001)
  4. Copy the data into a Google Sheet and import this into Mapped. Note: this can take hours...
  5. Create a map from this data source, configure it to show the population data for each area, then check that zooming in and out switches between MSOA, LSOA and Output Area shapes

Caveats

  • The front end logic is not clean – main challenge was getting the loading indicator to work with fetchMore(). This ultimately forced me to bring the useDataByBoundary hook up to a higher component level.
  • There's more global state now because the active tileset (and AnalyticalAreaType) needs to be known by multiple components. Before, this value was fixed and saved on the report config.
  • I have updated the import GenericData logic to save the Point() for all area types, as a generic solution for querying data within map bounds.
  • There is more state in PoliticalChoropleths now to cache the fill property of the choropleth layer. This reduces flicker but perhaps there is a more elegant way to do it.

* Command to import output areas and MSOA/LSOA
* GraphQL query for choropleth data can be filtered by map bounds
* Front-end automatically selects the tileset for the zoom level
  (census boundary type only)
@commonknowledge-bot commonknowledge-bot deployed to feat/census-data-maps - meep-database PR #188 January 20, 2025 15:58 — with Render Active
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/census-data-maps - meep-intelligence-hub-backend PR #188 January 20, 2025 15:58 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/census-data-maps - meep-intelligence-hub-backend PR #188 January 20, 2025 15:58 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/census-data-maps - meep-intelligence-hub-frontend PR #188 January 20, 2025 15:58 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot deployed to feat/census-data-maps - meep-intelligence-hub-worker PR #188 January 20, 2025 15:58 — with Render Active
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/census-data-maps - meep-intelligence-hub-backend PR #188 January 20, 2025 15:58 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/census-data-maps - meep-intelligence-hub-frontend PR #188 January 20, 2025 15:58 — with Render Destroyed
@janbaykara
Copy link
Member

This is looking and working fairly well. Some things I'm noticing so far:

Geometry resolution might be too poor right now

LSOA boundaries seem not to be aligned to MSOAs, but presumably they are aggregates. Are our Mapbox geometries too low resolution and can we beef them up? I wonder if we want to use geojson files for them as we did for OAs. The same might be sayable for MSOAs but I can't actually see because they're pretty small. I would say not to worry about the zoom levels for now though.

Screenshot 2025-01-21 at 17 41 31

Aggregation seems a bit skew-wiff

This might be a choropleth API related issue but these aggregations don't make sense to me:

Looks like 024 and 024D are aggregating the same, though they are evidently different levels of geography. Is there an area querying issue here or something?

Addressing that might help to reveal if numbers are adding/summing correctly.

Dataset is https://www.nomisweb.co.uk/output/census/2021/census2021-ts013.zip

Screenshot 2025-01-21 at 17 41 55 Screenshot 2025-01-21 at 17 41 38 Screenshot 2025-01-21 at 17 42 09

@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/census-data-maps - meep-intelligence-hub-backend PR #188 January 22, 2025 15:16 — with Render Destroyed
@joaquimds joaquimds force-pushed the feat/census-data-maps branch from 179e508 to bd11fc4 Compare January 22, 2025 15:16
@commonknowledge-bot commonknowledge-bot deployed to feat/census-data-maps - meep-intelligence-hub-worker PR #188 January 22, 2025 15:17 — with Render Active
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/census-data-maps - meep-intelligence-hub-worker PR #188 January 22, 2025 17:33 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/census-data-maps - meep-intelligence-hub-backend PR #188 January 22, 2025 17:33 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/census-data-maps - meep-intelligence-hub-frontend PR #188 January 22, 2025 17:33 — with Render Destroyed
@joaquimds joaquimds force-pushed the feat/census-data-maps branch from 98436df to 68a32a8 Compare January 22, 2025 17:35
@commonknowledge-bot commonknowledge-bot deployed to feat/census-data-maps - meep-intelligence-hub-worker PR #188 January 22, 2025 17:35 — with Render Active
@joaquimds joaquimds force-pushed the feat/census-data-maps branch from 68a32a8 to 2a74185 Compare January 22, 2025 17:45
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/census-data-maps - meep-intelligence-hub-frontend PR #188 January 22, 2025 17:45 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/census-data-maps - meep-intelligence-hub-worker PR #188 January 22, 2025 17:45 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/census-data-maps - meep-intelligence-hub-backend PR #188 January 22, 2025 17:45 — with Render Destroyed
@joaquimds joaquimds merged commit 2104e67 into main Jan 22, 2025
4 checks passed
@joaquimds joaquimds deleted the feat/census-data-maps branch January 22, 2025 17:53
Copy link

sentry-io bot commented Jan 23, 2025

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ GraphQLError: name 'firstsecond' is not defined /graphql View Issue

Did you find this useful? React with a 👍 or 👎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants