From d479ed35d8f368599189755c878cb93d5989d943 Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Thu, 23 Oct 2025 14:32:24 +0100 Subject: [PATCH 01/15] fix: refine test environment logging and config - Sets log level via environment variable with sensible default - Initialises JSON logger only for debug log level to reduce noise - Adds option to configure backend service URL with environment variable - Improves log tagging for better traceability - Suppresses verbose asset request logs in test runs --- config/environments/test.rb | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/config/environments/test.rb b/config/environments/test.rb index 981ac8a..75fe216 100644 --- a/config/environments/test.rb +++ b/config/environments/test.rb @@ -36,14 +36,22 @@ # Print deprecation notices to the stderr. config.active_support.deprecation = :stderr + # Don't print a log message every time an asset file is loaded + config.assets.quiet = true + + # Set the log level to the value of the LOG_LEVEL environment variable, or 'info' if not set + config.log_level = ENV.fetch('LOG_LEVEL', 'info').to_sym + # Tag rails logs with useful information config.log_tags = %i[subdomain request_id request_method] + # When sync mode is true, all output is immediately flushed to the underlying # operating system and is not buffered by Ruby internally. $stdout.sync = true - # Log the stdout output to the Epimorphics JSON logging gem - config.logger = JsonRailsLogger::Logger.new($stdout) - # Set the log level to the value of the LOG_LEVEL environment variable, or 'info' if not set - config.log_level = ENV.fetch('LOG_LEVEL', 'info').to_sym + # Log the stdout output to the Epimorphics JSON logging gem + config.logger = JsonRailsLogger::Logger.new($stdout) if config.log_level == :debug + # API location can be specified in the environment + # But defaults to the dev service + config.api_service_url = ENV.fetch('API_SERVICE_URL', 'http://localhost:8888') end From 6991f05eb0fdd20d26b6a46b1030024a144ebb79 Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Thu, 23 Oct 2025 14:32:38 +0100 Subject: [PATCH 02/15] chore: update bundler version in lockfile - Bumps lockfile bundler version for improved compatibility - Helps ensure consistent dependency management across environments --- Gemfile.lock | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Gemfile.lock b/Gemfile.lock index d60bbd7..0590d53 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -474,4 +474,4 @@ DEPENDENCIES web-console BUNDLED WITH - 2.7.1 + 2.7.2 From a0facb145f544a1a647b5e8bd0df93d1452d731b Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Thu, 23 Oct 2025 15:05:37 +0100 Subject: [PATCH 03/15] chore: replace and update the changelog - Replaces the manually maintained changelog with an automated one following a structured format - Updates recent release notes with improved formatting and greater detail - Structures the changelog for easier tracking of significant updates and version history --- CHANGELOG.md | 252 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 155 insertions(+), 97 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index eee3bfe..c5675ff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,26 +1,43 @@ -# Change log for HMLR landing page app +# Changelog -This app presents the landing page experience for landregistry.data.gov.uk, -including the SPARQL Qonsole +All notable changes to this project will be documented in this file. -## Unreleased +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## 2.2.3 - 2025-09 +## [Unreleased] -- Upates accessibility statements in English and Welsh [180](https://github.com/epimorphics/lr-landing/issues/180) +## [2.2.3] - 2025-09 -## 2.2.2 - 2025-09 +### Changed -- Updated to use latest Qonsole-rails gem, v2.2.1, to resolve JSON response issue - [85](https://github.com/epimorphics/qonsole-rails/issues/85) +- Updated accessibility statements in English and Welsh + [#180](https://github.com/epimorphics/lr-landing/issues/180) -## 2.2.1 - 2025-08 +## [2.2.2] - 2025-09 + +### Changed + +- Updated to use latest Qonsole-rails gem, v2.2.1, to resolve JSON response + issue [#85](https://github.com/epimorphics/qonsole-rails/issues/85) + +## [2.2.1] - 2025-08 + +### Added -- Resolved incorrect link to PPD Detailed Documentation -- Updated LR Common Styles gem to continue to address security issues - Added logic to bypass pre-push checks when only markdown files are staged -## 2.2.0 - 2025-08 +### Changed + +- Updated LR Common Styles gem to continue to address security issues + +### Fixed + +- Resolved incorrect link to PPD Detailed Documentation + +## [2.2.0] - 2025-08 + +### Changed - Bumped rails and all related gems to version 8.0.2.1 - Upgraded rubocop and rubocop-rails to latest 1.x and 2.x releases @@ -29,113 +46,154 @@ including the SPARQL Qonsole other gems for security and compatibility - Synchronised dependency constraints across related gems -## 2.1.3 - 2025-07 +## [2.1.3] - 2025-07 + +### Changed - Updated `qonsole_rails` version to include latest improvements -- Upgrade dependency management to improve project stability -- Enhance Docker handling in scripts with refined command order -- Introduce documentation for git hooks to streamline onboarding -- Optimise makefile by removing redundant elements +- Upgraded dependency management to improve project stability +- Enhanced Docker handling in scripts with refined command order +- Optimised makefile by removing redundant elements + +### Added + +- Introduced documentation for git hooks to streamline onboarding + +## [2.1.2] - 2025-07 -## 2.1.2 - 2025-07 +### Changed - Updated various gems to new versions, including `qonsole_rails`, - `lr_common_styles`, and `reline`. + `lr_common_styles`, and `reline` - Notable updates include major upgrades in `faraday`, `rubocop`, `nokogiri`, - and dependencies such as `sentry-rails` and `thor`. -- Modified `.rubocop.yml` to adjust the plugin configuration syntax. -- Refactored integration tests to include language parameters in request - methods. + and dependencies such as `sentry-rails` and `thor` +- Modified `.rubocop.yml` to adjust the plugin configuration syntax +- Refactored integration tests to include language parameters in request methods -## 2.1.1 - 2025-06 +## [2.1.1] - 2025-06 + +### Added -- Optimizes Docker configuration by refactoring the Dockerfile for build - efficiency, setting default environment variables, and refining network - settings. -- Enhances logging by adding checks for environment variables and better logging - output for operational clarity. -- Updates dependencies to ensure compatibility and performance. - Introduced a helper module to dynamically display the current environment in the UI, defaulting to "development" when not set -## 2.1.0 - 2025-05 +### Changed + +- Optimised Docker configuration by refactoring the Dockerfile for build + efficiency, setting default environment variables, and refining network + settings +- Enhanced logging by adding checks for environment variables and better logging + output for operational clarity +- Updated dependencies to ensure compatibility and performance + +## [2.1.0] - 2025-05 + +### Added + +- Added structured JSON logging for improved observability +- Updated git hooks to ensure tests pass before commits and pushes + +### Changed + +- Refactored code and updated Rubocop configuration for better code quality and + consistency +- Updated Rails to v8.0.2 alongside respective dependencies -- Adds structured JSON logging for improved observability. -- Updates git hooks to ensure tests pass before commits and pushes. -- Refactors code and updates Rubocop configuration for better code quality and - consistency. -- Updates Rails to v8.0.2 alongside respective dependencies +## [2.0.6] - 2025-04 -## 2.0.6 - 2025-04 +### Changed - Updated the `Uglifier` gem to `Terser` to allow ES6 syntax in the codebase -- Creation of `.env.development` to define configurations such as API URL, port, - and Sentry settings for the development environment -- Introduction of a `.githooks/post-commit` script that builds a Docker image - after a successful commit. The script only triggers for branches that match - "issue", "spike", or "task" -- Modification of `.githooks/pre-commit` and `.githooks/pre-push` to prevent - checks on 'hotfix', 'rebase', or 'production' branches -- The `.gitignore` file is updated to ignore `.env` and `.env*.local` files, +- Modified `.githooks/pre-commit` and `.githooks/pre-push` to prevent checks on + 'hotfix', 'rebase', or 'production' branches +- Updated Gemfile and Gemfile.lock to new gem versions +- Modified the Makefile to use the `AWS_REGION` variable when constructing the + ECR repository URL, making it region-aware +- Updated the `.gitignore` file to ignore `.env` and `.env*.local` files, excluding `.env.development` -- The Makefile is modified to use the `AWS_REGION` variable when constructing - the ECR repository URL, making it region-aware. -- Gemfile and Gemfile.lock updated to new gem versions -- A `Procfile.dev` was created to allow for local development with foreman or + +### Added + +- Created `.env.development` to define configurations such as API URL, port, and + Sentry settings for the development environment +- Introduced a `.githooks/post-commit` script that builds a Docker image after a + successful commit. The script only triggers for branches that match "issue", + "spike", or "task" +- Created a `Procfile.dev` to allow for local development with foreman or similar tools to start both the web and api processes -## 2.0.5 - 2025-03 +## [2.0.5] - 2025-03 + +### Added -- Improved error handling in application controller - Added pre-commit and pre-push hooks + +### Changed + +- Improved error handling in application controller - Updated locale handling in application controller - Improved Sentry configuration for different environments -## 2.0.4 - 2025-02 - -- (Jon) Updated log level configuration across environments - - Added log level setting based on environment variable. - - Default to 'debug' in development and 'info' in production/test. - [GH-63](https://github.com/epimorphics/lr-landing/issues/63) -- (Jon) Changed the way boot information is printed. -- (Jon) Switched to JSON format for better compatibility with logging services. -- (Jon) Refactored method to improve clarity and structure. -- (Jon) Made the version string immutable by freezing it -- (Jon) Included `puma-metrics` gem for better monitoring as now using Rails 6 - or greater -- (Jon) Updated Gemfile and Gemfile.lock to reflect the addition -- (Jon) Set up a configurable metrics port with a default value. -- (Jon) Updated the binding URL for the metrics server in development. -- (Jon) Removed old git reference for the qonsole_rails gem. -- (Jon) Added the qonsole_rails gem to the source block instead. -- (Jon) Updated several gems to their latest versions. -- (Jon) Cleaned up commented-out paths in the Gemfile. - -## 2.0.3 - 2025-01 - -- (Jon) Improves error metrics reporting to ensure that logging always happens - with the appropriate severity depending on the exception status while reducing - the types of errors that can trigger a an error metric and therefore a - notification in slack - [GH-149](https://github.com/epimorphics/hmlr-linked-data/issues/149) - -## 2.0.2 - 2024-12 - -- (Bogdan) Updated all gems, including `json_rails_logger` - -## 2.0.1 - 2024-10 - -- (Bogdan) Fixed an issue with CSS for the checkboxes in the Qonsole query form - -## 2.0.0 - 2024-10 - -- (Bogdan) Upgraded the `qonsole-rails` and `lr_common_styles` gems to latest - versions (which are now running on ruby `3.3.5` and rails `7.2.2` as well) -- (Bogdan) Upgraded alpine to `3.20` -- (Bogdan) Upgraded rails to `7.2.2` -- (Bogdan) Upgraded ruby to `3.3.5` -- (Jon) Removed the public/fees-caluclator.html file as it is no longer needed - [GH-140](https://github.com/epimorphics/lr-landing/issues/140) +## [2.0.4] - 2025-02 + +### Added + +- Included `puma-metrics` gem for better monitoring as now using Rails 6 or + greater +- Updated Gemfile and Gemfile.lock to reflect the addition +- Set up a configurable metrics port with a default value + +### Changed + +- Updated log level configuration across environments + [#63](https://github.com/epimorphics/lr-landing/issues/63) +- Changed the way boot information is printed +- Switched to JSON format for better compatibility with logging services +- Refactored method to improve clarity and structure +- Made the version string immutable by freezing it +- Updated several gems to their latest versions +- Cleaned up commented-out paths in the Gemfile +- Updated the binding URL for the metrics server in development +- Removed old git reference for the qonsole_rails gem +- Added the qonsole_rails gem to the source block instead + +## [2.0.3] - 2025-01 + +### Changed + +- Improved error metrics reporting to ensure that logging always happens with + the appropriate severity depending on the exception status while reducing the + types of errors that can trigger an error metric and therefore a notification + in slack [#149](https://github.com/epimorphics/hmlr-linked-data/issues/149) + +## [2.0.2] - 2024-12 + +### Changed + +- Updated all gems, including `json_rails_logger` + +## [2.0.1] - 2024-10 + +### Fixed + +- Fixed an issue with CSS for the checkboxes in the Qonsole query form + +## [2.0.0] - 2024-10 + +### Changed + +- Upgraded the `qonsole-rails` and `lr_common_styles` gems to latest versions + (which are now running on ruby `3.3.5` and rails `7.2.2` as well) +- Upgraded alpine to `3.20` +- Upgraded rails to `7.2.2` +- Upgraded ruby to `3.3.5` + +### Removed + +- Removed the public/fees-caluclator.html file as it is no longer needed + [#140](https://github.com/epimorphics/lr-landing/issues/140) + +--- ## 1.8.0 - 2024-09 From 004df7065b1740466917ab50197bd307ded502b7 Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Fri, 31 Oct 2025 17:56:07 +0000 Subject: [PATCH 04/15] chore: update dependencies for latest features - Upgrade core and third-party libraries to latest versions - Update lockfile to match new versions - Bump versions for bundled dependencies and update constraints - Maintain compatibility and bring in latest bug fixes --- Gemfile.lock | 212 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 123 insertions(+), 89 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 0590d53..5e46b87 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -10,29 +10,31 @@ GIT GEM remote: https://rubygems.org/ specs: - actioncable (8.0.2.1) - actionpack (= 8.0.2.1) - activesupport (= 8.0.2.1) + action_text-trix (2.1.15) + railties + actioncable (8.1.1) + actionpack (= 8.1.1) + activesupport (= 8.1.1) nio4r (~> 2.0) websocket-driver (>= 0.6.1) zeitwerk (~> 2.6) - actionmailbox (8.0.2.1) - actionpack (= 8.0.2.1) - activejob (= 8.0.2.1) - activerecord (= 8.0.2.1) - activestorage (= 8.0.2.1) - activesupport (= 8.0.2.1) + actionmailbox (8.1.1) + actionpack (= 8.1.1) + activejob (= 8.1.1) + activerecord (= 8.1.1) + activestorage (= 8.1.1) + activesupport (= 8.1.1) mail (>= 2.8.0) - actionmailer (8.0.2.1) - actionpack (= 8.0.2.1) - actionview (= 8.0.2.1) - activejob (= 8.0.2.1) - activesupport (= 8.0.2.1) + actionmailer (8.1.1) + actionpack (= 8.1.1) + actionview (= 8.1.1) + activejob (= 8.1.1) + activesupport (= 8.1.1) mail (>= 2.8.0) rails-dom-testing (~> 2.2) - actionpack (8.0.2.1) - actionview (= 8.0.2.1) - activesupport (= 8.0.2.1) + actionpack (8.1.1) + actionview (= 8.1.1) + activesupport (= 8.1.1) nokogiri (>= 1.8.5) rack (>= 2.2.4) rack-session (>= 1.0.1) @@ -40,42 +42,43 @@ GEM rails-dom-testing (~> 2.2) rails-html-sanitizer (~> 1.6) useragent (~> 0.16) - actiontext (8.0.2.1) - actionpack (= 8.0.2.1) - activerecord (= 8.0.2.1) - activestorage (= 8.0.2.1) - activesupport (= 8.0.2.1) + actiontext (8.1.1) + action_text-trix (~> 2.1.15) + actionpack (= 8.1.1) + activerecord (= 8.1.1) + activestorage (= 8.1.1) + activesupport (= 8.1.1) globalid (>= 0.6.0) nokogiri (>= 1.8.5) - actionview (8.0.2.1) - activesupport (= 8.0.2.1) + actionview (8.1.1) + activesupport (= 8.1.1) builder (~> 3.1) erubi (~> 1.11) rails-dom-testing (~> 2.2) rails-html-sanitizer (~> 1.6) - activejob (8.0.2.1) - activesupport (= 8.0.2.1) + activejob (8.1.1) + activesupport (= 8.1.1) globalid (>= 0.3.6) - activemodel (8.0.2.1) - activesupport (= 8.0.2.1) - activerecord (8.0.2.1) - activemodel (= 8.0.2.1) - activesupport (= 8.0.2.1) + activemodel (8.1.1) + activesupport (= 8.1.1) + activerecord (8.1.1) + activemodel (= 8.1.1) + activesupport (= 8.1.1) timeout (>= 0.4.0) - activestorage (8.0.2.1) - actionpack (= 8.0.2.1) - activejob (= 8.0.2.1) - activerecord (= 8.0.2.1) - activesupport (= 8.0.2.1) + activestorage (8.1.1) + actionpack (= 8.1.1) + activejob (= 8.1.1) + activerecord (= 8.1.1) + activesupport (= 8.1.1) marcel (~> 1.0) - activesupport (8.0.2.1) + activesupport (8.1.1) base64 - benchmark (>= 0.3) bigdecimal concurrent-ruby (~> 1.0, >= 1.3.1) connection_pool (>= 2.2.5) drb i18n (>= 1.6, < 2) + json logger (>= 1.4.2) minitest (>= 5.1) securerandom (>= 0.3) @@ -86,8 +89,9 @@ GEM execjs (~> 2) backport (1.2.0) base64 (0.3.0) - benchmark (0.4.1) - bigdecimal (3.2.2) + benchmark (0.5.0) + benchmark-ips (2.14.0) + bigdecimal (3.3.1) bindex (0.8.1) bootstrap-sass (3.4.1) autoprefixer-rails (>= 5.2.1) @@ -95,12 +99,30 @@ GEM builder (3.3.0) byebug (12.0.0) concurrent-ruby (1.3.5) - connection_pool (2.5.3) + connection_pool (2.5.4) crass (1.0.6) - date (3.4.1) + date (3.5.0) + derailed_benchmarks (2.2.1) + base64 + benchmark-ips (~> 2) + bigdecimal + drb + get_process_mem + heapy (~> 0) + logger + memory_profiler (>= 0, < 2) + mini_histogram (>= 0.3.0) + mutex_m + ostruct + rack (>= 1) + rack-test + rake (> 10, < 14) + ruby-statistics (>= 4.0.1) + ruby2_keywords + thor (>= 0.19, < 2) diff-lcs (1.6.2) drb (2.2.3) - erb (5.0.2) + erb (5.1.3) erubi (1.13.1) execjs (2.10.0) faraday (2.13.4) @@ -128,7 +150,7 @@ GEM get_process_mem (1.0.0) bigdecimal (>= 2.0) ffi (~> 1.0) - globalid (1.2.1) + globalid (1.3.0) activesupport (>= 6.1) govuk_elements_rails (3.0.2) govuk_frontend_toolkit (>= 5.2.0) @@ -142,11 +164,13 @@ GEM temple (>= 0.8.2) thor tilt - haml-rails (2.1.0) + haml-rails (3.0.0) actionpack (>= 5.1) activesupport (>= 5.1) haml (>= 4.0.6) railties (>= 5.1) + heapy (0.2.0) + thor http_accept_language (2.1.1) i18n (1.14.7) concurrent-ruby (~> 1.0) @@ -168,7 +192,7 @@ GEM rails-dom-testing (>= 1, < 3) railties (>= 4.2.0) thor (>= 0.14, < 2.0) - json (2.13.2) + json (2.15.2) kramdown (2.5.1) rexml (>= 3.3.9) kramdown-parser-gfm (1.1.0) @@ -186,20 +210,24 @@ GEM loofah (2.24.1) crass (~> 1.0.2) nokogiri (>= 1.12.0) - mail (2.8.1) + mail (2.9.0) + logger mini_mime (>= 0.1.1) net-imap net-pop net-smtp - marcel (1.0.4) + marcel (1.1.0) + memory_profiler (1.1.0) + mini_histogram (0.3.1) mini_mime (1.1.5) - minitest (5.25.5) + minitest (5.26.0) modernizr-rails (2.7.1) modulejs-rails (2.2.0.0) railties (>= 4.0) + mutex_m (0.3.0) net-http (0.6.0) uri - net-imap (0.5.9) + net-imap (0.5.12) date net-protocol net-pop (0.1.2) @@ -208,22 +236,22 @@ GEM timeout net-smtp (0.5.1) net-protocol - nio4r (2.7.4) - nokogiri (1.18.9-aarch64-linux-gnu) + nio4r (2.7.5) + nokogiri (1.18.10-aarch64-linux-gnu) racc (~> 1.4) - nokogiri (1.18.9-aarch64-linux-musl) + nokogiri (1.18.10-aarch64-linux-musl) racc (~> 1.4) - nokogiri (1.18.9-arm-linux-gnu) + nokogiri (1.18.10-arm-linux-gnu) racc (~> 1.4) - nokogiri (1.18.9-arm-linux-musl) + nokogiri (1.18.10-arm-linux-musl) racc (~> 1.4) - nokogiri (1.18.9-arm64-darwin) + nokogiri (1.18.10-arm64-darwin) racc (~> 1.4) - nokogiri (1.18.9-x86_64-darwin) + nokogiri (1.18.10-x86_64-darwin) racc (~> 1.4) - nokogiri (1.18.9-x86_64-linux-gnu) + nokogiri (1.18.10-x86_64-linux-gnu) racc (~> 1.4) - nokogiri (1.18.9-x86_64-linux-musl) + nokogiri (1.18.10-x86_64-linux-musl) racc (~> 1.4) observer (0.1.2) ostruct (0.6.3) @@ -231,7 +259,7 @@ GEM parser (3.3.9.0) ast (~> 2.4.1) racc - pp (0.6.2) + pp (0.6.3) prettyprint prettyprint (0.2.0) prism (1.4.0) @@ -246,7 +274,7 @@ GEM prometheus-client (>= 0.10) puma (>= 6.0) racc (1.8.1) - rack (3.2.0) + rack (3.2.3) rack-contrib (2.5.0) rack (< 4) rack-session (2.1.1) @@ -256,20 +284,20 @@ GEM rack (>= 1.3) rackup (2.2.1) rack (>= 3) - rails (8.0.2.1) - actioncable (= 8.0.2.1) - actionmailbox (= 8.0.2.1) - actionmailer (= 8.0.2.1) - actionpack (= 8.0.2.1) - actiontext (= 8.0.2.1) - actionview (= 8.0.2.1) - activejob (= 8.0.2.1) - activemodel (= 8.0.2.1) - activerecord (= 8.0.2.1) - activestorage (= 8.0.2.1) - activesupport (= 8.0.2.1) + rails (8.1.1) + actioncable (= 8.1.1) + actionmailbox (= 8.1.1) + actionmailer (= 8.1.1) + actionpack (= 8.1.1) + actiontext (= 8.1.1) + actionview (= 8.1.1) + activejob (= 8.1.1) + activemodel (= 8.1.1) + activerecord (= 8.1.1) + activestorage (= 8.1.1) + activesupport (= 8.1.1) bundler (>= 1.15.0) - railties (= 8.0.2.1) + railties (= 8.1.1) rails-dom-testing (2.3.0) activesupport (>= 5.0.0) minitest @@ -277,24 +305,26 @@ GEM rails-html-sanitizer (1.6.2) loofah (~> 2.21) nokogiri (>= 1.15.7, != 1.16.7, != 1.16.6, != 1.16.5, != 1.16.4, != 1.16.3, != 1.16.2, != 1.16.1, != 1.16.0.rc1, != 1.16.0) - railties (8.0.2.1) - actionpack (= 8.0.2.1) - activesupport (= 8.0.2.1) + railties (8.1.1) + actionpack (= 8.1.1) + activesupport (= 8.1.1) irb (~> 1.13) rackup (>= 1.0.0) rake (>= 12.2) thor (~> 1.0, >= 1.2.2) + tsort (>= 0.2) zeitwerk (~> 2.6) rainbow (3.1.1) - rake (13.3.0) + rake (13.3.1) rb-fsevent (0.11.2) rb-inotify (0.11.1) ffi (~> 1.0) rbs (3.6.1) logger - rdoc (6.14.2) + rdoc (6.15.0) erb psych (>= 4.0.0) + tsort regexp_parser (2.11.2) reline (0.6.2) io-console (~> 0.5) @@ -328,6 +358,8 @@ GEM prism (>= 1.2, < 2.0) rbs (>= 3, < 5) ruby-progressbar (1.13.0) + ruby-statistics (4.1.0) + ruby2_keywords (0.0.5) sass (3.7.4) sass-listen (~> 4.0.0) sass-listen (4.0.0) @@ -378,19 +410,21 @@ GEM actionpack (>= 6.1) activesupport (>= 6.1) sprockets (>= 3.0.0) + stackprof (0.2.27) stringio (3.1.7) temple (0.10.3) terser (1.2.6) execjs (>= 0.3.0, < 3) thor (1.4.0) tilt (2.6.1) - timeout (0.4.3) + timeout (0.4.4) + tsort (0.2.0) tzinfo (2.0.6) concurrent-ruby (~> 1.0) unicode-display_width (3.1.5) unicode-emoji (~> 4.0, >= 4.0.4) unicode-emoji (4.0.4) - uri (1.0.3) + uri (1.1.0) useragent (0.16.11) web-console (4.2.1) actionview (>= 6.0.0) @@ -409,30 +443,30 @@ GEM GEM remote: https://rubygems.pkg.github.com/epimorphics/ specs: - json_rails_logger (2.0.6) + json_rails_logger (2.2.0) json lograge railties - lr_common_styles (2.3.0) + lr_common_styles (2.3.1) bootstrap-sass (~> 3.4.1) font-awesome-rails (~> 4.7.0) govuk_elements_rails (= 3.0.2) govuk_frontend_toolkit (~> 9.0) govuk_template (~> 0.26.0) - haml-rails (~> 2.1) + haml-rails (~> 3.0) jquery-rails (~> 4.6) lodash-rails (~> 4.17) modernizr-rails (~> 2.7) modulejs-rails (~> 2.2.0) - rails (~> 8.0.0) + rails (~> 8.0) sass-rails (~> 6.0) - qonsole_rails (2.2.1) + qonsole_rails (2.3.0) faraday (~> 2.13) - faraday-encoding (~> 0.0.6) - faraday-follow_redirects (~> 0.3.0) + faraday-encoding (~> 0.0, >= 0.0.6) + faraday-follow_redirects (~> 0.3, >= 0.3.0) faraday-retry (~> 2.0) font-awesome-rails (~> 4.7.0) - haml-rails (~> 2.1) + haml-rails (~> 3.0) jquery-datatables-rails (~> 3.4) jquery-rails (~> 4.6) lodash-rails (~> 4.17) From 9ec53911707baa7699ab79ba55d0a94802657a0d Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Fri, 31 Oct 2025 17:56:08 +0000 Subject: [PATCH 05/15] feat: add benchmarking and profiling tools - Add development tools for performance benchmarking - Ensure profiling libraries and helper utilities are available - Update environment for related dependencies --- Gemfile | 2 ++ Gemfile.lock | 2 ++ 2 files changed, 4 insertions(+) diff --git a/Gemfile b/Gemfile index 81435d7..5041a99 100644 --- a/Gemfile +++ b/Gemfile @@ -32,6 +32,8 @@ end group :development do gem 'ruby-lsp' gem 'solargraph' + gem 'derailed_benchmarks' + gem 'stackprof' # Devtools panel for Rails development - loading from the GitHub repo # (https://github.com/dejan/rails_panel/issues/209#issuecomment-2621877079_) gem 'meta_request', github: 'dejan/rails_panel', ref: 'meta_request-v0.8.5' diff --git a/Gemfile.lock b/Gemfile.lock index 5e46b87..dd68003 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -485,6 +485,7 @@ PLATFORMS DEPENDENCIES byebug + derailed_benchmarks get_process_mem haml-rails http_accept_language @@ -504,6 +505,7 @@ DEPENDENCIES sass-rails sentry-rails solargraph + stackprof terser web-console From b7ada36d7794c7419eecef5d79d49fa82cdc65fd Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Fri, 31 Oct 2025 17:56:08 +0000 Subject: [PATCH 06/15] chore: add system tools for resource introspection - Include system utility required for process monitoring in container - Prepare for resource usage logging methods --- Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/Dockerfile b/Dockerfile index 4fbae55..720c1fd 100644 --- a/Dockerfile +++ b/Dockerfile @@ -12,6 +12,7 @@ RUN apk add --update --no-cache \ git \ nodejs \ tzdata \ + procps \ && gem update --system # for Bundler From 4a726b8b69486a1e88622ad018bd3b1b7c7d4635 Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Fri, 31 Oct 2025 17:56:08 +0000 Subject: [PATCH 07/15] refactor: improve error handling and resource logging - Update exception rescue paths to clarify outcomes - Add hooks and helpers for internal resource usage logging - Enhance logging for error reporting and debugging - Use standard naming and status resolution for HTTP errors - Tidy unused methods and add references for clarity --- app/controllers/application_controller.rb | 52 ++++++++++++++++++----- 1 file changed, 42 insertions(+), 10 deletions(-) diff --git a/app/controllers/application_controller.rb b/app/controllers/application_controller.rb index 2063ff4..41b4a70 100644 --- a/app/controllers/application_controller.rb +++ b/app/controllers/application_controller.rb @@ -8,6 +8,7 @@ class ApplicationController < ActionController::Base # For APIs, you may want to use :null_session instead. protect_from_forgery with: :exception, prepend: true + # before_action :log_resource_usage before_action :set_locale before_action :change_default_caching_policy around_action :log_response @@ -58,14 +59,15 @@ def log_response # or attempt to render a generic error page if no specific error page exists unless Rails.application.config.consider_all_requests_local rescue_from StandardError do |e| + puts "[ApplicationController] Caught exception: #{e.class}: #{e.message}" # Trigger the appropriate error handling method based on the exception case e.class when ActionController::RoutingError, ActionView::MissingTemplate - :render404 + :render_404 when ActionController::InvalidCrossOriginRequest - :render403 + :render_403 when ActionController::BadRequest, ActionController::ParameterMissing - :render400 + :render_400 else :handle_internal_error end @@ -81,17 +83,17 @@ def handle_internal_error(exception) # rubocop:disable Metrics/MethodLength when UnprocessableEntity render_error(422) else - cname = exception.class.name logged_fields = { status: Rack::Utils::HTTP_STATUS_CODES[exception] } - if Rails.env.development? || Rails.logger.debug? + if Rails.logger.debug? logged_fields[:backtrace] = exception.backtrace.join("\n") end Log.error( - "No explicit error page for exception #{exception} - #{cname}", + "No explicit error page for #{exception.class.name} - #{exception}", logged_fields ) + # Instrument ActiveSupport::Notifications for internal errors but only for 500 errors: sentry_code = instrument_application_error(exception) render_error(500, sentry_code) @@ -117,11 +119,11 @@ def render_500(_exception = nil) # rubocop:disable Naming/VariableNumber def render_error(status, sentry_code = nil) reset_response - error_status = Rack::Utils::SYMBOL_TO_STATUS_CODE[status] if status.is_a?(Symbol) + status = Rack::Utils::SYMBOL_TO_STATUS_CODE[status] if status.is_a?(Symbol) respond_to do |format| - format.html { render_html_error_page(error_status, sentry_code) } + format.html { render_html_error_page(status, sentry_code) } # Anything else returns the status as human readable plain string - format.all { render plain: Rack::Utils::HTTP_STATUS_CODES[status].to_s, status: error_status } + format.all { render plain: Rack::Utils::HTTP_STATUS_CODES[status].to_s, status: status } end end @@ -142,6 +144,8 @@ def version private + #! UNUSED METHOD - kept for reference + # Set the Sentry user context for error tracking def set_sentry_user return unless signed_in? && Rails.env.production? @@ -168,10 +172,38 @@ def instrument_application_error(exc) # rubocop:disable Metrics/CyclomaticComple end # Log the exception to the Rails logger with the appropriate severity Rails.logger.send(err[:status] < 500 ? :warn : :error, JSON.generate(err)) - # Return unless the status code is 500 or greater to ensure subscribers are NOT notified + # Return unless the status code is 500 or greater to ensure metrics subscribers are NOT notified return unless err[:status] >= 500 # Instrument the internal error event to notify subscribers of the error ActiveSupport::Notifications.instrument('internal_error.application', exception: err) end + + # Get the current process resource usage using the `ps` command + # -o format ~ User-defined format. format is a single argument in the + # form of a blank-separated or comma-separated list, which + # offers a way to specify individual output columns. + # psr: processor number last used + # etime: elapsed time since the process started + # pcpu: percentage of CPU used by the process + # pmem: percentage of memory used by the process + # rss: resident set size (physical memory used) in kilobytes + # vsz: virtual memory size in kilobytes + # -p pid ~ Select by process ID + # Source: https://discuss.rubyonrails.org/t/how-to-reduce-memory-footprint-of-a-rails-app/39388/6 + # Reference: https://man7.org/linux/man-pages/man1/ps.1.html + def log_resource_usage + if Rails.env.production? # && Rails.logger.debug? + x,psr,etime,pcpu,pmem,rss,vsz = `ps -o psr,etime,pcpu,pmem,rss,vsz -p #{Process.pid}`&.split('\n')[1]&.split(/\s+/) + resources = { psr:, etime:, pcpu:, pmem:, rss:, vsz: } + Rails.logger.info( + "***DEBUG: resource_usage: + rails_env=#{Rails.env} + pid=#{Process.pid} + #{resources.compact!.map { |k, v| "#{k}=#{v}" }.join(' ')} + req_method=#{request.method} + req_uri=#{request.url} + ") + end + end end From 58f19a002c28d9d13a2064dae7568fc779cf6411 Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Fri, 31 Oct 2025 17:56:08 +0000 Subject: [PATCH 08/15] fix: handle missing status in metrics subscriber - Use safe dig to handle nil or missing status - Prevent exceptions during metrics emission --- app/subscribers/api_prometheus_subscriber.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/subscribers/api_prometheus_subscriber.rb b/app/subscribers/api_prometheus_subscriber.rb index 09c1e9e..df9dcd9 100644 --- a/app/subscribers/api_prometheus_subscriber.rb +++ b/app/subscribers/api_prometheus_subscriber.rb @@ -7,7 +7,7 @@ class ApiPrometheusSubscriber < ActiveSupport::Subscriber def response(event) response = event.payload[:response] duration = event.payload[:duration] - status = response.status.to_s + status = response&.dig(:status).to_s Prometheus::Client.registry .get(:api_status) From 5bae76bd3c28ba0d47d7bac04f1c7f4520fef68a Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Fri, 31 Oct 2025 17:56:08 +0000 Subject: [PATCH 09/15] chore: tidy logging configuration in development - Standardise placement of log level configuration in environment - Add explanatory comment for configuration block - Remove redundant configuration for clarity --- config/environments/development.rb | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/config/environments/development.rb b/config/environments/development.rb index 2f58e50..51f4e21 100644 --- a/config/environments/development.rb +++ b/config/environments/development.rb @@ -38,6 +38,8 @@ # Don't print a log message every time an asset file is loaded config.assets.quiet = true + # Set the log level to the value of the LOG_LEVEL environment variable, or 'debug' if not set + config.log_level = ENV.fetch('LOG_LEVEL', 'debug').to_sym # Tag rails logs with useful information config.log_tags = %i[subdomain request_id request_method] # When sync mode is true, all output is immediately flushed to the underlying @@ -55,7 +57,4 @@ # API location can be specified in the environment but defaults to the dev service # Here we are still providing the API_SERVICE_URL for qonsole config.api_service_url = ENV.fetch('API_SERVICE_URL', 'http://localhost:8888') - - # Set the log level to the value of the LOG_LEVEL environment variable, or 'debug' if not set - config.log_level = ENV.fetch('LOG_LEVEL', 'debug').to_sym end From ad9f72f9946d102ddd0d6d641d88326546df5ef0 Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Sat, 1 Nov 2025 01:30:55 +0000 Subject: [PATCH 10/15] chore: bump version and update changelog - Bumped version number for new release - Added summary of changes, improvements and fixes - Highlighted security, performance, and logging enhancements - Listed fixed issue related to missing status in metrics --- CHANGELOG.md | 22 ++++++++++++++++++++++ app/lib/version.rb | 2 +- 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c5675ff..ce83e34 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [2.2.4] - 2025-11 + +### Added + +- Benchmarking and profiling tools were added for performance analysis +- System tools for resource introspection were added +- Hooks and helpers for internal resource usage logging were added + +### Changed + +- Core and third-party libraries were upgraded to latest versions for security + and stability +- Dependency constraints were updated and bundled dependencies were bumped +- Logging configuration in development environment was standardised and tidied +- Exception rescue paths were updated to clarify outcomes +- Error handling and resource logging were improved + +### Fixed + +- Missing status in metrics subscriber was handled to prevent exceptions during + metrics emission + ## [2.2.3] - 2025-09 ### Changed diff --git a/app/lib/version.rb b/app/lib/version.rb index c7fdb88..b8800ad 100644 --- a/app/lib/version.rb +++ b/app/lib/version.rb @@ -3,7 +3,7 @@ module Version MAJOR = 2 MINOR = 2 - PATCH = 3 + PATCH = 4 SUFFIX = nil VERSION = "#{MAJOR}.#{MINOR}.#{PATCH}#{SUFFIX && ".#{SUFFIX}"}".freeze end From 4851e2fddd2093fcc0d237700aa27c171cd827cf Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Mon, 3 Nov 2025 09:22:11 +0000 Subject: [PATCH 11/15] fix: replace curly quotes with html entities - Replaces all curly single quotes with numeric HTML entities for consistency - Ensures correct display of single quotation marks in web browsers - Updates all affected documentation text for clarity --- app/views/doc/ppd.html.haml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/app/views/doc/ppd.html.haml b/app/views/doc/ppd.html.haml index f986c73..4d5cf3f 100644 --- a/app/views/doc/ppd.html.haml +++ b/app/views/doc/ppd.html.haml @@ -92,7 +92,7 @@ A standard PPD transaction concerns a single residential property sold for value to private individual. An additional PPD transaction concerns repossessions, buy-to-lets and transfers of property sold to non-private individuals. %p - The conversion process used takes these items and generates a number of ‘triples’ that are uploaded to a triple store that’s made available using a cloud based triple store. Descriptions of linked data formats are widely described in various web resources and are not covered here, HM Land Registry generate SPARQL update (.ru) files each month and these are described below. + The conversion process used takes these items and generates a number of ‘triples’ that are uploaded to a triple store that’s made available using a cloud based triple store. Descriptions of linked data formats are widely described in various web resources and are not covered here, HM Land Registry generate SPARQL update (.ru) files each month and these are described below. %p To describe a transaction we have the following information: %p @@ -317,7 +317,7 @@ %code ppi:publishDate property. This has since been removed. %p - The record status was part of the first release of price paid data and has since been made obsolete but remains in the store so that queries that included it will still run. All transactions will have a record status of ‘add’. + The record status was part of the first release of price paid data and has since been made obsolete but remains in the store so that queries that included it will still run. All transactions will have a record status of ‘add’. %p The triples generated for this transaction are: %pre @@ -510,7 +510,7 @@ Which reveals that there are 9 entries that satisfy that query. A query for the county of Derbyshire reveals that there are 177,264 transactions (both queries correct as of June 2014). Note that for some queries the count function has been disabled due to the size of the dataset, this is particularly relevant for queries at the transaction-record level, for example this query will return new build properties in Derbyshire adding the count function will result in a warning message.: %pre %a{:href => "http://landregistry.data.gov.uk/data/ppi/transaction-record?&propertyAddress.county=DERBYSHIRE&newBuild=false"} http://landregistry.data.gov.uk/data/ppi/transaction-record?&propertyAddress.county=DERBYSHIRE&newBuild=false -%h3 Example SPARQL query using ‘filter’ +%h3 Example SPARQL query using ‘filter’ %p This is an example of a custom SPARQL query that we used. The customer request was to use a SPARQL query to find property where part of the PAON was Alveston within postcode CV37 7AE. This can be easily done using the Price Paid report Builder but using SPARQL can be more flexible when dealing with multiple PAONs. @@ -581,7 +581,7 @@ ?item ppd:propertyAddress/lrcommon:paon ?ppd_propertyAddressPaon FILTER regex(?ppd_propertyAddressPaon,'alve','i') %p - These lines identify PAON as part of the select and that the result should be filtered to find the string ‘alve’ anywhere within the PAON, the third argument specifies that the string is not case sensitive. + These lines identify PAON as part of the select and that the result should be filtered to find the string ‘alve’ anywhere within the PAON, the third argument specifies that the string is not case sensitive. %p This gives a WHERE clause that looks like this and, when run, will return a single property transaction for Alveston Cottage %pre @@ -605,7 +605,7 @@ %p More than one string can be searched for using %code || - to separate each argument, to return PAONs that contain the string ‘alve’ or the number 5 (stored as a string not numeric): + to separate each argument, to return PAONs that contain the string ‘alve’ or the number 5 (stored as a string not numeric): %pre :preserve ?item ppd:propertyAddress/lrcommon:paon ?ppd_propertyAddressPaon From 17bcd78b22b91f59c95b170846f08c779b4e4d2c Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Mon, 3 Nov 2025 09:22:12 +0000 Subject: [PATCH 12/15] chore: update Qonsole link - Resolves old link path for SPARQL query page [#163](https://github.com/epimorphics/lr-landing/issues/163) --- app/views/doc/ppd.html.haml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/views/doc/ppd.html.haml b/app/views/doc/ppd.html.haml index 4d5cf3f..f44a4dc 100644 --- a/app/views/doc/ppd.html.haml +++ b/app/views/doc/ppd.html.haml @@ -478,7 +478,7 @@ %p Alongside the report builder we have created a SPARQL query page that can be found here = succeed "." do - %a{:href => "/app/qonsole"} http://landregistry.data.gov.uk/app/hpi/qonsole + %a{:href => "/app/qonsole"} http://landregistry.data.gov.uk/app/qonsole %p We also have made a %a{:href => "https://www.w3.org/wiki/SparqlEndpoints"} SPARQL endpoint From cef26f521b91c77b3166df9aa78908e5cb856913 Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Mon, 3 Nov 2025 09:22:12 +0000 Subject: [PATCH 13/15] docs: update description for sparql query example - Changes example heading to use HTML entity instead of curly quote - Keeps terminology consistent with rest of document examples --- app/views/doc/ppd.html.haml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/views/doc/ppd.html.haml b/app/views/doc/ppd.html.haml index f44a4dc..42cbaff 100644 --- a/app/views/doc/ppd.html.haml +++ b/app/views/doc/ppd.html.haml @@ -489,7 +489,7 @@ Included at the end of this document is an example of a SPARQL query using FILTER to restrict the results returned. %h3 Via the API %p - It is possible to query the Price Paid Dataset directly through the API, this largely removes the need to know how to use SPARQL as long as the structure of the data is known. Using the example transaction above we know that there is a street called ‘Pear Tree Street’ and that street is part of the address within the PPI data structure. With that information we can enter this URL into a web browser + It is possible to query the Price Paid Dataset directly through the API, this largely removes the need to know how to use SPARQL as long as the structure of the data is known. Using the example transaction above we know that there is a street called ‘Pear Tree Street’ and that street is part of the address within the PPI data structure. With that information we can enter this URL into a web browser %pre %a{:href => "http://landregistry.data.gov.uk/data/ppi/address?&street=PEAR%20TREE%20STREET"} http://landregistry.data.gov.uk/data/ppi/address?&street=PEAR%20TREE%20STREET %p From 0551891fc96e43132aadc60b83dff8b56d0732a8 Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Mon, 3 Nov 2025 16:27:12 +0000 Subject: [PATCH 14/15] refactor: remove deprecated timezone config option - Cleans up obsolete timezone preservation setting - Prepares codebase for compatibility with future framework updates --- config/application.rb | 6 ------ 1 file changed, 6 deletions(-) diff --git a/config/application.rb b/config/application.rb index faf885a..a3c21dd 100644 --- a/config/application.rb +++ b/config/application.rb @@ -39,12 +39,6 @@ class Application < Rails::Application # Run "rake -D time" for a list of tasks for finding time zone names. Default is UTC. # config.time_zone = 'Central Time (US & Canada)' - # !DEPRECATION WARNING: `to_time` will always preserve the full system - # ! timezone offset rather than offset of the receiver in Rails 8.1. - # * To opt in to the new behavior and maintain system timezone, - # * set the value to :zone; otherwise, set to false - config.active_support.to_time_preserves_timezone = :zone - # The default locale is :en and all translations from config/locales/*.rb,yml are auto loaded. # config.i18n.load_path += Dir[Rails.root.join('my', 'locales', '*.{rb,yml}').to_s] config.i18n.available_locales = %i[en cy] From cd15a54cb95e2103f582d9b7e817a88c33ab05d4 Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Mon, 3 Nov 2025 16:29:12 +0000 Subject: [PATCH 15/15] docs: Updated CHANGELOG - Drops outdated configuration for future framework support - Prepares for new version upgrade by cleaning legacy setting --- CHANGELOG.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index ce83e34..a247c0f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Exception rescue paths were updated to clarify outcomes - Error handling and resource logging were improved +### Removed + +- Removed deprecated `to_time_preserves_timezone` configuration for Rails 8.1+ + compatibility + ### Fixed - Missing status in metrics subscriber was handled to prevent exceptions during