diff --git a/CHANGELOG.md b/CHANGELOG.md index eee3bfe..a247c0f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,26 +1,70 @@ -# Change log for HMLR landing page app +# Changelog -This app presents the landing page experience for landregistry.data.gov.uk, -including the SPARQL Qonsole +All notable changes to this project will be documented in this file. -## Unreleased +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## 2.2.3 - 2025-09 +## [Unreleased] -- Upates accessibility statements in English and Welsh [180](https://github.com/epimorphics/lr-landing/issues/180) +## [2.2.4] - 2025-11 -## 2.2.2 - 2025-09 +### Added -- Updated to use latest Qonsole-rails gem, v2.2.1, to resolve JSON response issue - [85](https://github.com/epimorphics/qonsole-rails/issues/85) +- Benchmarking and profiling tools were added for performance analysis +- System tools for resource introspection were added +- Hooks and helpers for internal resource usage logging were added -## 2.2.1 - 2025-08 +### Changed + +- Core and third-party libraries were upgraded to latest versions for security + and stability +- Dependency constraints were updated and bundled dependencies were bumped +- Logging configuration in development environment was standardised and tidied +- Exception rescue paths were updated to clarify outcomes +- Error handling and resource logging were improved + +### Removed + +- Removed deprecated `to_time_preserves_timezone` configuration for Rails 8.1+ + compatibility + +### Fixed + +- Missing status in metrics subscriber was handled to prevent exceptions during + metrics emission + +## [2.2.3] - 2025-09 + +### Changed + +- Updated accessibility statements in English and Welsh + [#180](https://github.com/epimorphics/lr-landing/issues/180) + +## [2.2.2] - 2025-09 + +### Changed + +- Updated to use latest Qonsole-rails gem, v2.2.1, to resolve JSON response + issue [#85](https://github.com/epimorphics/qonsole-rails/issues/85) + +## [2.2.1] - 2025-08 + +### Added -- Resolved incorrect link to PPD Detailed Documentation -- Updated LR Common Styles gem to continue to address security issues - Added logic to bypass pre-push checks when only markdown files are staged -## 2.2.0 - 2025-08 +### Changed + +- Updated LR Common Styles gem to continue to address security issues + +### Fixed + +- Resolved incorrect link to PPD Detailed Documentation + +## [2.2.0] - 2025-08 + +### Changed - Bumped rails and all related gems to version 8.0.2.1 - Upgraded rubocop and rubocop-rails to latest 1.x and 2.x releases @@ -29,113 +73,154 @@ including the SPARQL Qonsole other gems for security and compatibility - Synchronised dependency constraints across related gems -## 2.1.3 - 2025-07 +## [2.1.3] - 2025-07 + +### Changed - Updated `qonsole_rails` version to include latest improvements -- Upgrade dependency management to improve project stability -- Enhance Docker handling in scripts with refined command order -- Introduce documentation for git hooks to streamline onboarding -- Optimise makefile by removing redundant elements +- Upgraded dependency management to improve project stability +- Enhanced Docker handling in scripts with refined command order +- Optimised makefile by removing redundant elements + +### Added -## 2.1.2 - 2025-07 +- Introduced documentation for git hooks to streamline onboarding + +## [2.1.2] - 2025-07 + +### Changed - Updated various gems to new versions, including `qonsole_rails`, - `lr_common_styles`, and `reline`. + `lr_common_styles`, and `reline` - Notable updates include major upgrades in `faraday`, `rubocop`, `nokogiri`, - and dependencies such as `sentry-rails` and `thor`. -- Modified `.rubocop.yml` to adjust the plugin configuration syntax. -- Refactored integration tests to include language parameters in request - methods. + and dependencies such as `sentry-rails` and `thor` +- Modified `.rubocop.yml` to adjust the plugin configuration syntax +- Refactored integration tests to include language parameters in request methods -## 2.1.1 - 2025-06 +## [2.1.1] - 2025-06 + +### Added -- Optimizes Docker configuration by refactoring the Dockerfile for build - efficiency, setting default environment variables, and refining network - settings. -- Enhances logging by adding checks for environment variables and better logging - output for operational clarity. -- Updates dependencies to ensure compatibility and performance. - Introduced a helper module to dynamically display the current environment in the UI, defaulting to "development" when not set -## 2.1.0 - 2025-05 +### Changed + +- Optimised Docker configuration by refactoring the Dockerfile for build + efficiency, setting default environment variables, and refining network + settings +- Enhanced logging by adding checks for environment variables and better logging + output for operational clarity +- Updated dependencies to ensure compatibility and performance + +## [2.1.0] - 2025-05 -- Adds structured JSON logging for improved observability. -- Updates git hooks to ensure tests pass before commits and pushes. -- Refactors code and updates Rubocop configuration for better code quality and - consistency. -- Updates Rails to v8.0.2 alongside respective dependencies +### Added -## 2.0.6 - 2025-04 +- Added structured JSON logging for improved observability +- Updated git hooks to ensure tests pass before commits and pushes + +### Changed + +- Refactored code and updated Rubocop configuration for better code quality and + consistency +- Updated Rails to v8.0.2 alongside respective dependencies + +## [2.0.6] - 2025-04 + +### Changed - Updated the `Uglifier` gem to `Terser` to allow ES6 syntax in the codebase -- Creation of `.env.development` to define configurations such as API URL, port, - and Sentry settings for the development environment -- Introduction of a `.githooks/post-commit` script that builds a Docker image - after a successful commit. The script only triggers for branches that match - "issue", "spike", or "task" -- Modification of `.githooks/pre-commit` and `.githooks/pre-push` to prevent - checks on 'hotfix', 'rebase', or 'production' branches -- The `.gitignore` file is updated to ignore `.env` and `.env*.local` files, +- Modified `.githooks/pre-commit` and `.githooks/pre-push` to prevent checks on + 'hotfix', 'rebase', or 'production' branches +- Updated Gemfile and Gemfile.lock to new gem versions +- Modified the Makefile to use the `AWS_REGION` variable when constructing the + ECR repository URL, making it region-aware +- Updated the `.gitignore` file to ignore `.env` and `.env*.local` files, excluding `.env.development` -- The Makefile is modified to use the `AWS_REGION` variable when constructing - the ECR repository URL, making it region-aware. -- Gemfile and Gemfile.lock updated to new gem versions -- A `Procfile.dev` was created to allow for local development with foreman or + +### Added + +- Created `.env.development` to define configurations such as API URL, port, and + Sentry settings for the development environment +- Introduced a `.githooks/post-commit` script that builds a Docker image after a + successful commit. The script only triggers for branches that match "issue", + "spike", or "task" +- Created a `Procfile.dev` to allow for local development with foreman or similar tools to start both the web and api processes -## 2.0.5 - 2025-03 +## [2.0.5] - 2025-03 + +### Added -- Improved error handling in application controller - Added pre-commit and pre-push hooks + +### Changed + +- Improved error handling in application controller - Updated locale handling in application controller - Improved Sentry configuration for different environments -## 2.0.4 - 2025-02 - -- (Jon) Updated log level configuration across environments - - Added log level setting based on environment variable. - - Default to 'debug' in development and 'info' in production/test. - [GH-63](https://github.com/epimorphics/lr-landing/issues/63) -- (Jon) Changed the way boot information is printed. -- (Jon) Switched to JSON format for better compatibility with logging services. -- (Jon) Refactored method to improve clarity and structure. -- (Jon) Made the version string immutable by freezing it -- (Jon) Included `puma-metrics` gem for better monitoring as now using Rails 6 - or greater -- (Jon) Updated Gemfile and Gemfile.lock to reflect the addition -- (Jon) Set up a configurable metrics port with a default value. -- (Jon) Updated the binding URL for the metrics server in development. -- (Jon) Removed old git reference for the qonsole_rails gem. -- (Jon) Added the qonsole_rails gem to the source block instead. -- (Jon) Updated several gems to their latest versions. -- (Jon) Cleaned up commented-out paths in the Gemfile. - -## 2.0.3 - 2025-01 - -- (Jon) Improves error metrics reporting to ensure that logging always happens - with the appropriate severity depending on the exception status while reducing - the types of errors that can trigger a an error metric and therefore a - notification in slack - [GH-149](https://github.com/epimorphics/hmlr-linked-data/issues/149) - -## 2.0.2 - 2024-12 - -- (Bogdan) Updated all gems, including `json_rails_logger` - -## 2.0.1 - 2024-10 - -- (Bogdan) Fixed an issue with CSS for the checkboxes in the Qonsole query form - -## 2.0.0 - 2024-10 - -- (Bogdan) Upgraded the `qonsole-rails` and `lr_common_styles` gems to latest - versions (which are now running on ruby `3.3.5` and rails `7.2.2` as well) -- (Bogdan) Upgraded alpine to `3.20` -- (Bogdan) Upgraded rails to `7.2.2` -- (Bogdan) Upgraded ruby to `3.3.5` -- (Jon) Removed the public/fees-caluclator.html file as it is no longer needed - [GH-140](https://github.com/epimorphics/lr-landing/issues/140) +## [2.0.4] - 2025-02 + +### Added + +- Included `puma-metrics` gem for better monitoring as now using Rails 6 or + greater +- Updated Gemfile and Gemfile.lock to reflect the addition +- Set up a configurable metrics port with a default value + +### Changed + +- Updated log level configuration across environments + [#63](https://github.com/epimorphics/lr-landing/issues/63) +- Changed the way boot information is printed +- Switched to JSON format for better compatibility with logging services +- Refactored method to improve clarity and structure +- Made the version string immutable by freezing it +- Updated several gems to their latest versions +- Cleaned up commented-out paths in the Gemfile +- Updated the binding URL for the metrics server in development +- Removed old git reference for the qonsole_rails gem +- Added the qonsole_rails gem to the source block instead + +## [2.0.3] - 2025-01 + +### Changed + +- Improved error metrics reporting to ensure that logging always happens with + the appropriate severity depending on the exception status while reducing the + types of errors that can trigger an error metric and therefore a notification + in slack [#149](https://github.com/epimorphics/hmlr-linked-data/issues/149) + +## [2.0.2] - 2024-12 + +### Changed + +- Updated all gems, including `json_rails_logger` + +## [2.0.1] - 2024-10 + +### Fixed + +- Fixed an issue with CSS for the checkboxes in the Qonsole query form + +## [2.0.0] - 2024-10 + +### Changed + +- Upgraded the `qonsole-rails` and `lr_common_styles` gems to latest versions + (which are now running on ruby `3.3.5` and rails `7.2.2` as well) +- Upgraded alpine to `3.20` +- Upgraded rails to `7.2.2` +- Upgraded ruby to `3.3.5` + +### Removed + +- Removed the public/fees-caluclator.html file as it is no longer needed + [#140](https://github.com/epimorphics/lr-landing/issues/140) + +--- ## 1.8.0 - 2024-09 diff --git a/Dockerfile b/Dockerfile index 4fbae55..720c1fd 100644 --- a/Dockerfile +++ b/Dockerfile @@ -12,6 +12,7 @@ RUN apk add --update --no-cache \ git \ nodejs \ tzdata \ + procps \ && gem update --system # for Bundler diff --git a/Gemfile b/Gemfile index 81435d7..5041a99 100644 --- a/Gemfile +++ b/Gemfile @@ -32,6 +32,8 @@ end group :development do gem 'ruby-lsp' gem 'solargraph' + gem 'derailed_benchmarks' + gem 'stackprof' # Devtools panel for Rails development - loading from the GitHub repo # (https://github.com/dejan/rails_panel/issues/209#issuecomment-2621877079_) gem 'meta_request', github: 'dejan/rails_panel', ref: 'meta_request-v0.8.5' diff --git a/Gemfile.lock b/Gemfile.lock index d60bbd7..dd68003 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -10,29 +10,31 @@ GIT GEM remote: https://rubygems.org/ specs: - actioncable (8.0.2.1) - actionpack (= 8.0.2.1) - activesupport (= 8.0.2.1) + action_text-trix (2.1.15) + railties + actioncable (8.1.1) + actionpack (= 8.1.1) + activesupport (= 8.1.1) nio4r (~> 2.0) websocket-driver (>= 0.6.1) zeitwerk (~> 2.6) - actionmailbox (8.0.2.1) - actionpack (= 8.0.2.1) - activejob (= 8.0.2.1) - activerecord (= 8.0.2.1) - activestorage (= 8.0.2.1) - activesupport (= 8.0.2.1) + actionmailbox (8.1.1) + actionpack (= 8.1.1) + activejob (= 8.1.1) + activerecord (= 8.1.1) + activestorage (= 8.1.1) + activesupport (= 8.1.1) mail (>= 2.8.0) - actionmailer (8.0.2.1) - actionpack (= 8.0.2.1) - actionview (= 8.0.2.1) - activejob (= 8.0.2.1) - activesupport (= 8.0.2.1) + actionmailer (8.1.1) + actionpack (= 8.1.1) + actionview (= 8.1.1) + activejob (= 8.1.1) + activesupport (= 8.1.1) mail (>= 2.8.0) rails-dom-testing (~> 2.2) - actionpack (8.0.2.1) - actionview (= 8.0.2.1) - activesupport (= 8.0.2.1) + actionpack (8.1.1) + actionview (= 8.1.1) + activesupport (= 8.1.1) nokogiri (>= 1.8.5) rack (>= 2.2.4) rack-session (>= 1.0.1) @@ -40,42 +42,43 @@ GEM rails-dom-testing (~> 2.2) rails-html-sanitizer (~> 1.6) useragent (~> 0.16) - actiontext (8.0.2.1) - actionpack (= 8.0.2.1) - activerecord (= 8.0.2.1) - activestorage (= 8.0.2.1) - activesupport (= 8.0.2.1) + actiontext (8.1.1) + action_text-trix (~> 2.1.15) + actionpack (= 8.1.1) + activerecord (= 8.1.1) + activestorage (= 8.1.1) + activesupport (= 8.1.1) globalid (>= 0.6.0) nokogiri (>= 1.8.5) - actionview (8.0.2.1) - activesupport (= 8.0.2.1) + actionview (8.1.1) + activesupport (= 8.1.1) builder (~> 3.1) erubi (~> 1.11) rails-dom-testing (~> 2.2) rails-html-sanitizer (~> 1.6) - activejob (8.0.2.1) - activesupport (= 8.0.2.1) + activejob (8.1.1) + activesupport (= 8.1.1) globalid (>= 0.3.6) - activemodel (8.0.2.1) - activesupport (= 8.0.2.1) - activerecord (8.0.2.1) - activemodel (= 8.0.2.1) - activesupport (= 8.0.2.1) + activemodel (8.1.1) + activesupport (= 8.1.1) + activerecord (8.1.1) + activemodel (= 8.1.1) + activesupport (= 8.1.1) timeout (>= 0.4.0) - activestorage (8.0.2.1) - actionpack (= 8.0.2.1) - activejob (= 8.0.2.1) - activerecord (= 8.0.2.1) - activesupport (= 8.0.2.1) + activestorage (8.1.1) + actionpack (= 8.1.1) + activejob (= 8.1.1) + activerecord (= 8.1.1) + activesupport (= 8.1.1) marcel (~> 1.0) - activesupport (8.0.2.1) + activesupport (8.1.1) base64 - benchmark (>= 0.3) bigdecimal concurrent-ruby (~> 1.0, >= 1.3.1) connection_pool (>= 2.2.5) drb i18n (>= 1.6, < 2) + json logger (>= 1.4.2) minitest (>= 5.1) securerandom (>= 0.3) @@ -86,8 +89,9 @@ GEM execjs (~> 2) backport (1.2.0) base64 (0.3.0) - benchmark (0.4.1) - bigdecimal (3.2.2) + benchmark (0.5.0) + benchmark-ips (2.14.0) + bigdecimal (3.3.1) bindex (0.8.1) bootstrap-sass (3.4.1) autoprefixer-rails (>= 5.2.1) @@ -95,12 +99,30 @@ GEM builder (3.3.0) byebug (12.0.0) concurrent-ruby (1.3.5) - connection_pool (2.5.3) + connection_pool (2.5.4) crass (1.0.6) - date (3.4.1) + date (3.5.0) + derailed_benchmarks (2.2.1) + base64 + benchmark-ips (~> 2) + bigdecimal + drb + get_process_mem + heapy (~> 0) + logger + memory_profiler (>= 0, < 2) + mini_histogram (>= 0.3.0) + mutex_m + ostruct + rack (>= 1) + rack-test + rake (> 10, < 14) + ruby-statistics (>= 4.0.1) + ruby2_keywords + thor (>= 0.19, < 2) diff-lcs (1.6.2) drb (2.2.3) - erb (5.0.2) + erb (5.1.3) erubi (1.13.1) execjs (2.10.0) faraday (2.13.4) @@ -128,7 +150,7 @@ GEM get_process_mem (1.0.0) bigdecimal (>= 2.0) ffi (~> 1.0) - globalid (1.2.1) + globalid (1.3.0) activesupport (>= 6.1) govuk_elements_rails (3.0.2) govuk_frontend_toolkit (>= 5.2.0) @@ -142,11 +164,13 @@ GEM temple (>= 0.8.2) thor tilt - haml-rails (2.1.0) + haml-rails (3.0.0) actionpack (>= 5.1) activesupport (>= 5.1) haml (>= 4.0.6) railties (>= 5.1) + heapy (0.2.0) + thor http_accept_language (2.1.1) i18n (1.14.7) concurrent-ruby (~> 1.0) @@ -168,7 +192,7 @@ GEM rails-dom-testing (>= 1, < 3) railties (>= 4.2.0) thor (>= 0.14, < 2.0) - json (2.13.2) + json (2.15.2) kramdown (2.5.1) rexml (>= 3.3.9) kramdown-parser-gfm (1.1.0) @@ -186,20 +210,24 @@ GEM loofah (2.24.1) crass (~> 1.0.2) nokogiri (>= 1.12.0) - mail (2.8.1) + mail (2.9.0) + logger mini_mime (>= 0.1.1) net-imap net-pop net-smtp - marcel (1.0.4) + marcel (1.1.0) + memory_profiler (1.1.0) + mini_histogram (0.3.1) mini_mime (1.1.5) - minitest (5.25.5) + minitest (5.26.0) modernizr-rails (2.7.1) modulejs-rails (2.2.0.0) railties (>= 4.0) + mutex_m (0.3.0) net-http (0.6.0) uri - net-imap (0.5.9) + net-imap (0.5.12) date net-protocol net-pop (0.1.2) @@ -208,22 +236,22 @@ GEM timeout net-smtp (0.5.1) net-protocol - nio4r (2.7.4) - nokogiri (1.18.9-aarch64-linux-gnu) + nio4r (2.7.5) + nokogiri (1.18.10-aarch64-linux-gnu) racc (~> 1.4) - nokogiri (1.18.9-aarch64-linux-musl) + nokogiri (1.18.10-aarch64-linux-musl) racc (~> 1.4) - nokogiri (1.18.9-arm-linux-gnu) + nokogiri (1.18.10-arm-linux-gnu) racc (~> 1.4) - nokogiri (1.18.9-arm-linux-musl) + nokogiri (1.18.10-arm-linux-musl) racc (~> 1.4) - nokogiri (1.18.9-arm64-darwin) + nokogiri (1.18.10-arm64-darwin) racc (~> 1.4) - nokogiri (1.18.9-x86_64-darwin) + nokogiri (1.18.10-x86_64-darwin) racc (~> 1.4) - nokogiri (1.18.9-x86_64-linux-gnu) + nokogiri (1.18.10-x86_64-linux-gnu) racc (~> 1.4) - nokogiri (1.18.9-x86_64-linux-musl) + nokogiri (1.18.10-x86_64-linux-musl) racc (~> 1.4) observer (0.1.2) ostruct (0.6.3) @@ -231,7 +259,7 @@ GEM parser (3.3.9.0) ast (~> 2.4.1) racc - pp (0.6.2) + pp (0.6.3) prettyprint prettyprint (0.2.0) prism (1.4.0) @@ -246,7 +274,7 @@ GEM prometheus-client (>= 0.10) puma (>= 6.0) racc (1.8.1) - rack (3.2.0) + rack (3.2.3) rack-contrib (2.5.0) rack (< 4) rack-session (2.1.1) @@ -256,20 +284,20 @@ GEM rack (>= 1.3) rackup (2.2.1) rack (>= 3) - rails (8.0.2.1) - actioncable (= 8.0.2.1) - actionmailbox (= 8.0.2.1) - actionmailer (= 8.0.2.1) - actionpack (= 8.0.2.1) - actiontext (= 8.0.2.1) - actionview (= 8.0.2.1) - activejob (= 8.0.2.1) - activemodel (= 8.0.2.1) - activerecord (= 8.0.2.1) - activestorage (= 8.0.2.1) - activesupport (= 8.0.2.1) + rails (8.1.1) + actioncable (= 8.1.1) + actionmailbox (= 8.1.1) + actionmailer (= 8.1.1) + actionpack (= 8.1.1) + actiontext (= 8.1.1) + actionview (= 8.1.1) + activejob (= 8.1.1) + activemodel (= 8.1.1) + activerecord (= 8.1.1) + activestorage (= 8.1.1) + activesupport (= 8.1.1) bundler (>= 1.15.0) - railties (= 8.0.2.1) + railties (= 8.1.1) rails-dom-testing (2.3.0) activesupport (>= 5.0.0) minitest @@ -277,24 +305,26 @@ GEM rails-html-sanitizer (1.6.2) loofah (~> 2.21) nokogiri (>= 1.15.7, != 1.16.7, != 1.16.6, != 1.16.5, != 1.16.4, != 1.16.3, != 1.16.2, != 1.16.1, != 1.16.0.rc1, != 1.16.0) - railties (8.0.2.1) - actionpack (= 8.0.2.1) - activesupport (= 8.0.2.1) + railties (8.1.1) + actionpack (= 8.1.1) + activesupport (= 8.1.1) irb (~> 1.13) rackup (>= 1.0.0) rake (>= 12.2) thor (~> 1.0, >= 1.2.2) + tsort (>= 0.2) zeitwerk (~> 2.6) rainbow (3.1.1) - rake (13.3.0) + rake (13.3.1) rb-fsevent (0.11.2) rb-inotify (0.11.1) ffi (~> 1.0) rbs (3.6.1) logger - rdoc (6.14.2) + rdoc (6.15.0) erb psych (>= 4.0.0) + tsort regexp_parser (2.11.2) reline (0.6.2) io-console (~> 0.5) @@ -328,6 +358,8 @@ GEM prism (>= 1.2, < 2.0) rbs (>= 3, < 5) ruby-progressbar (1.13.0) + ruby-statistics (4.1.0) + ruby2_keywords (0.0.5) sass (3.7.4) sass-listen (~> 4.0.0) sass-listen (4.0.0) @@ -378,19 +410,21 @@ GEM actionpack (>= 6.1) activesupport (>= 6.1) sprockets (>= 3.0.0) + stackprof (0.2.27) stringio (3.1.7) temple (0.10.3) terser (1.2.6) execjs (>= 0.3.0, < 3) thor (1.4.0) tilt (2.6.1) - timeout (0.4.3) + timeout (0.4.4) + tsort (0.2.0) tzinfo (2.0.6) concurrent-ruby (~> 1.0) unicode-display_width (3.1.5) unicode-emoji (~> 4.0, >= 4.0.4) unicode-emoji (4.0.4) - uri (1.0.3) + uri (1.1.0) useragent (0.16.11) web-console (4.2.1) actionview (>= 6.0.0) @@ -409,30 +443,30 @@ GEM GEM remote: https://rubygems.pkg.github.com/epimorphics/ specs: - json_rails_logger (2.0.6) + json_rails_logger (2.2.0) json lograge railties - lr_common_styles (2.3.0) + lr_common_styles (2.3.1) bootstrap-sass (~> 3.4.1) font-awesome-rails (~> 4.7.0) govuk_elements_rails (= 3.0.2) govuk_frontend_toolkit (~> 9.0) govuk_template (~> 0.26.0) - haml-rails (~> 2.1) + haml-rails (~> 3.0) jquery-rails (~> 4.6) lodash-rails (~> 4.17) modernizr-rails (~> 2.7) modulejs-rails (~> 2.2.0) - rails (~> 8.0.0) + rails (~> 8.0) sass-rails (~> 6.0) - qonsole_rails (2.2.1) + qonsole_rails (2.3.0) faraday (~> 2.13) - faraday-encoding (~> 0.0.6) - faraday-follow_redirects (~> 0.3.0) + faraday-encoding (~> 0.0, >= 0.0.6) + faraday-follow_redirects (~> 0.3, >= 0.3.0) faraday-retry (~> 2.0) font-awesome-rails (~> 4.7.0) - haml-rails (~> 2.1) + haml-rails (~> 3.0) jquery-datatables-rails (~> 3.4) jquery-rails (~> 4.6) lodash-rails (~> 4.17) @@ -451,6 +485,7 @@ PLATFORMS DEPENDENCIES byebug + derailed_benchmarks get_process_mem haml-rails http_accept_language @@ -470,8 +505,9 @@ DEPENDENCIES sass-rails sentry-rails solargraph + stackprof terser web-console BUNDLED WITH - 2.7.1 + 2.7.2 diff --git a/app/controllers/application_controller.rb b/app/controllers/application_controller.rb index 2063ff4..41b4a70 100644 --- a/app/controllers/application_controller.rb +++ b/app/controllers/application_controller.rb @@ -8,6 +8,7 @@ class ApplicationController < ActionController::Base # For APIs, you may want to use :null_session instead. protect_from_forgery with: :exception, prepend: true + # before_action :log_resource_usage before_action :set_locale before_action :change_default_caching_policy around_action :log_response @@ -58,14 +59,15 @@ def log_response # or attempt to render a generic error page if no specific error page exists unless Rails.application.config.consider_all_requests_local rescue_from StandardError do |e| + puts "[ApplicationController] Caught exception: #{e.class}: #{e.message}" # Trigger the appropriate error handling method based on the exception case e.class when ActionController::RoutingError, ActionView::MissingTemplate - :render404 + :render_404 when ActionController::InvalidCrossOriginRequest - :render403 + :render_403 when ActionController::BadRequest, ActionController::ParameterMissing - :render400 + :render_400 else :handle_internal_error end @@ -81,17 +83,17 @@ def handle_internal_error(exception) # rubocop:disable Metrics/MethodLength when UnprocessableEntity render_error(422) else - cname = exception.class.name logged_fields = { status: Rack::Utils::HTTP_STATUS_CODES[exception] } - if Rails.env.development? || Rails.logger.debug? + if Rails.logger.debug? logged_fields[:backtrace] = exception.backtrace.join("\n") end Log.error( - "No explicit error page for exception #{exception} - #{cname}", + "No explicit error page for #{exception.class.name} - #{exception}", logged_fields ) + # Instrument ActiveSupport::Notifications for internal errors but only for 500 errors: sentry_code = instrument_application_error(exception) render_error(500, sentry_code) @@ -117,11 +119,11 @@ def render_500(_exception = nil) # rubocop:disable Naming/VariableNumber def render_error(status, sentry_code = nil) reset_response - error_status = Rack::Utils::SYMBOL_TO_STATUS_CODE[status] if status.is_a?(Symbol) + status = Rack::Utils::SYMBOL_TO_STATUS_CODE[status] if status.is_a?(Symbol) respond_to do |format| - format.html { render_html_error_page(error_status, sentry_code) } + format.html { render_html_error_page(status, sentry_code) } # Anything else returns the status as human readable plain string - format.all { render plain: Rack::Utils::HTTP_STATUS_CODES[status].to_s, status: error_status } + format.all { render plain: Rack::Utils::HTTP_STATUS_CODES[status].to_s, status: status } end end @@ -142,6 +144,8 @@ def version private + #! UNUSED METHOD - kept for reference + # Set the Sentry user context for error tracking def set_sentry_user return unless signed_in? && Rails.env.production? @@ -168,10 +172,38 @@ def instrument_application_error(exc) # rubocop:disable Metrics/CyclomaticComple end # Log the exception to the Rails logger with the appropriate severity Rails.logger.send(err[:status] < 500 ? :warn : :error, JSON.generate(err)) - # Return unless the status code is 500 or greater to ensure subscribers are NOT notified + # Return unless the status code is 500 or greater to ensure metrics subscribers are NOT notified return unless err[:status] >= 500 # Instrument the internal error event to notify subscribers of the error ActiveSupport::Notifications.instrument('internal_error.application', exception: err) end + + # Get the current process resource usage using the `ps` command + # -o format ~ User-defined format. format is a single argument in the + # form of a blank-separated or comma-separated list, which + # offers a way to specify individual output columns. + # psr: processor number last used + # etime: elapsed time since the process started + # pcpu: percentage of CPU used by the process + # pmem: percentage of memory used by the process + # rss: resident set size (physical memory used) in kilobytes + # vsz: virtual memory size in kilobytes + # -p pid ~ Select by process ID + # Source: https://discuss.rubyonrails.org/t/how-to-reduce-memory-footprint-of-a-rails-app/39388/6 + # Reference: https://man7.org/linux/man-pages/man1/ps.1.html + def log_resource_usage + if Rails.env.production? # && Rails.logger.debug? + x,psr,etime,pcpu,pmem,rss,vsz = `ps -o psr,etime,pcpu,pmem,rss,vsz -p #{Process.pid}`&.split('\n')[1]&.split(/\s+/) + resources = { psr:, etime:, pcpu:, pmem:, rss:, vsz: } + Rails.logger.info( + "***DEBUG: resource_usage: + rails_env=#{Rails.env} + pid=#{Process.pid} + #{resources.compact!.map { |k, v| "#{k}=#{v}" }.join(' ')} + req_method=#{request.method} + req_uri=#{request.url} + ") + end + end end diff --git a/app/lib/version.rb b/app/lib/version.rb index c7fdb88..b8800ad 100644 --- a/app/lib/version.rb +++ b/app/lib/version.rb @@ -3,7 +3,7 @@ module Version MAJOR = 2 MINOR = 2 - PATCH = 3 + PATCH = 4 SUFFIX = nil VERSION = "#{MAJOR}.#{MINOR}.#{PATCH}#{SUFFIX && ".#{SUFFIX}"}".freeze end diff --git a/app/subscribers/api_prometheus_subscriber.rb b/app/subscribers/api_prometheus_subscriber.rb index 09c1e9e..df9dcd9 100644 --- a/app/subscribers/api_prometheus_subscriber.rb +++ b/app/subscribers/api_prometheus_subscriber.rb @@ -7,7 +7,7 @@ class ApiPrometheusSubscriber < ActiveSupport::Subscriber def response(event) response = event.payload[:response] duration = event.payload[:duration] - status = response.status.to_s + status = response&.dig(:status).to_s Prometheus::Client.registry .get(:api_status) diff --git a/app/views/doc/ppd.html.haml b/app/views/doc/ppd.html.haml index f986c73..42cbaff 100644 --- a/app/views/doc/ppd.html.haml +++ b/app/views/doc/ppd.html.haml @@ -92,7 +92,7 @@ A standard PPD transaction concerns a single residential property sold for value to private individual. An additional PPD transaction concerns repossessions, buy-to-lets and transfers of property sold to non-private individuals. %p - The conversion process used takes these items and generates a number of ‘triples’ that are uploaded to a triple store that’s made available using a cloud based triple store. Descriptions of linked data formats are widely described in various web resources and are not covered here, HM Land Registry generate SPARQL update (.ru) files each month and these are described below. + The conversion process used takes these items and generates a number of ‘triples’ that are uploaded to a triple store that’s made available using a cloud based triple store. Descriptions of linked data formats are widely described in various web resources and are not covered here, HM Land Registry generate SPARQL update (.ru) files each month and these are described below. %p To describe a transaction we have the following information: %p @@ -317,7 +317,7 @@ %code ppi:publishDate property. This has since been removed. %p - The record status was part of the first release of price paid data and has since been made obsolete but remains in the store so that queries that included it will still run. All transactions will have a record status of ‘add’. + The record status was part of the first release of price paid data and has since been made obsolete but remains in the store so that queries that included it will still run. All transactions will have a record status of ‘add’. %p The triples generated for this transaction are: %pre @@ -478,7 +478,7 @@ %p Alongside the report builder we have created a SPARQL query page that can be found here = succeed "." do - %a{:href => "/app/qonsole"} http://landregistry.data.gov.uk/app/hpi/qonsole + %a{:href => "/app/qonsole"} http://landregistry.data.gov.uk/app/qonsole %p We also have made a %a{:href => "https://www.w3.org/wiki/SparqlEndpoints"} SPARQL endpoint @@ -489,7 +489,7 @@ Included at the end of this document is an example of a SPARQL query using FILTER to restrict the results returned. %h3 Via the API %p - It is possible to query the Price Paid Dataset directly through the API, this largely removes the need to know how to use SPARQL as long as the structure of the data is known. Using the example transaction above we know that there is a street called ‘Pear Tree Street’ and that street is part of the address within the PPI data structure. With that information we can enter this URL into a web browser + It is possible to query the Price Paid Dataset directly through the API, this largely removes the need to know how to use SPARQL as long as the structure of the data is known. Using the example transaction above we know that there is a street called ‘Pear Tree Street’ and that street is part of the address within the PPI data structure. With that information we can enter this URL into a web browser %pre %a{:href => "http://landregistry.data.gov.uk/data/ppi/address?&street=PEAR%20TREE%20STREET"} http://landregistry.data.gov.uk/data/ppi/address?&street=PEAR%20TREE%20STREET %p @@ -510,7 +510,7 @@ Which reveals that there are 9 entries that satisfy that query. A query for the county of Derbyshire reveals that there are 177,264 transactions (both queries correct as of June 2014). Note that for some queries the count function has been disabled due to the size of the dataset, this is particularly relevant for queries at the transaction-record level, for example this query will return new build properties in Derbyshire adding the count function will result in a warning message.: %pre %a{:href => "http://landregistry.data.gov.uk/data/ppi/transaction-record?&propertyAddress.county=DERBYSHIRE&newBuild=false"} http://landregistry.data.gov.uk/data/ppi/transaction-record?&propertyAddress.county=DERBYSHIRE&newBuild=false -%h3 Example SPARQL query using ‘filter’ +%h3 Example SPARQL query using ‘filter’ %p This is an example of a custom SPARQL query that we used. The customer request was to use a SPARQL query to find property where part of the PAON was Alveston within postcode CV37 7AE. This can be easily done using the Price Paid report Builder but using SPARQL can be more flexible when dealing with multiple PAONs. @@ -581,7 +581,7 @@ ?item ppd:propertyAddress/lrcommon:paon ?ppd_propertyAddressPaon FILTER regex(?ppd_propertyAddressPaon,'alve','i') %p - These lines identify PAON as part of the select and that the result should be filtered to find the string ‘alve’ anywhere within the PAON, the third argument specifies that the string is not case sensitive. + These lines identify PAON as part of the select and that the result should be filtered to find the string ‘alve’ anywhere within the PAON, the third argument specifies that the string is not case sensitive. %p This gives a WHERE clause that looks like this and, when run, will return a single property transaction for Alveston Cottage %pre @@ -605,7 +605,7 @@ %p More than one string can be searched for using %code || - to separate each argument, to return PAONs that contain the string ‘alve’ or the number 5 (stored as a string not numeric): + to separate each argument, to return PAONs that contain the string ‘alve’ or the number 5 (stored as a string not numeric): %pre :preserve ?item ppd:propertyAddress/lrcommon:paon ?ppd_propertyAddressPaon diff --git a/config/application.rb b/config/application.rb index faf885a..a3c21dd 100644 --- a/config/application.rb +++ b/config/application.rb @@ -39,12 +39,6 @@ class Application < Rails::Application # Run "rake -D time" for a list of tasks for finding time zone names. Default is UTC. # config.time_zone = 'Central Time (US & Canada)' - # !DEPRECATION WARNING: `to_time` will always preserve the full system - # ! timezone offset rather than offset of the receiver in Rails 8.1. - # * To opt in to the new behavior and maintain system timezone, - # * set the value to :zone; otherwise, set to false - config.active_support.to_time_preserves_timezone = :zone - # The default locale is :en and all translations from config/locales/*.rb,yml are auto loaded. # config.i18n.load_path += Dir[Rails.root.join('my', 'locales', '*.{rb,yml}').to_s] config.i18n.available_locales = %i[en cy] diff --git a/config/environments/development.rb b/config/environments/development.rb index 2f58e50..51f4e21 100644 --- a/config/environments/development.rb +++ b/config/environments/development.rb @@ -38,6 +38,8 @@ # Don't print a log message every time an asset file is loaded config.assets.quiet = true + # Set the log level to the value of the LOG_LEVEL environment variable, or 'debug' if not set + config.log_level = ENV.fetch('LOG_LEVEL', 'debug').to_sym # Tag rails logs with useful information config.log_tags = %i[subdomain request_id request_method] # When sync mode is true, all output is immediately flushed to the underlying @@ -55,7 +57,4 @@ # API location can be specified in the environment but defaults to the dev service # Here we are still providing the API_SERVICE_URL for qonsole config.api_service_url = ENV.fetch('API_SERVICE_URL', 'http://localhost:8888') - - # Set the log level to the value of the LOG_LEVEL environment variable, or 'debug' if not set - config.log_level = ENV.fetch('LOG_LEVEL', 'debug').to_sym end diff --git a/config/environments/test.rb b/config/environments/test.rb index 981ac8a..75fe216 100644 --- a/config/environments/test.rb +++ b/config/environments/test.rb @@ -36,14 +36,22 @@ # Print deprecation notices to the stderr. config.active_support.deprecation = :stderr + # Don't print a log message every time an asset file is loaded + config.assets.quiet = true + + # Set the log level to the value of the LOG_LEVEL environment variable, or 'info' if not set + config.log_level = ENV.fetch('LOG_LEVEL', 'info').to_sym + # Tag rails logs with useful information config.log_tags = %i[subdomain request_id request_method] + # When sync mode is true, all output is immediately flushed to the underlying # operating system and is not buffered by Ruby internally. $stdout.sync = true - # Log the stdout output to the Epimorphics JSON logging gem - config.logger = JsonRailsLogger::Logger.new($stdout) - # Set the log level to the value of the LOG_LEVEL environment variable, or 'info' if not set - config.log_level = ENV.fetch('LOG_LEVEL', 'info').to_sym + # Log the stdout output to the Epimorphics JSON logging gem + config.logger = JsonRailsLogger::Logger.new($stdout) if config.log_level == :debug + # API location can be specified in the environment + # But defaults to the dev service + config.api_service_url = ENV.fetch('API_SERVICE_URL', 'http://localhost:8888') end