26-09-2025

Automated Database Migrations & Project Health Updates

This release introduces a significant architectural enhancement with a new automated database migration system to handle schema evolution. It also includes substantial improvements to project documentation and general dependency maintenance.

🗄️ Automated Database Migration System

A generic, versioned database migration system for MongoDB has been implemented. The rationale for this system is to provide a robust, automated, and traceable way to apply schema changes on application startup, ensuring data consistency across all deployments without manual intervention.

• Introduced a new migration system that automatically detects and applies pending schema changes to the MongoDB database on application startup. [#62]
• Migrations are versioned using their Pull Request merge date and include the PR ID and summary, providing direct traceability for every schema change. [#62]
• The system is designed to be idempotent, meaning migrations can be safely run multiple times without causing errors or data inconsistencies. [#62]
• The first migration has been implemented to refactor the adConfig structure within RemoteConfig documents to a more flexible, role-based model. [#62]

📝 Documentation & Project Health

This release includes several updates focused on improving the developer experience, code clarity, and long-term project maintainability.

• The main README.md file has been significantly restructured for better readability, featuring collapsible sections to showcase API features in a more organized way. [#59]
• Updated project dependencies, including upgrading very_good_analysis to 9.0.0 and pinning several git dependencies to specific commit hashes for build consistency. [#61]
• Added explicit type annotations to several providers and configuration files to improve code clarity and maintainability. [#61]

18-09-2025

Flexible Data Route Access & Non-Destructive Seeding

This release introduces a major enhancement to the API's flexibility by enabling the generic data route to serve public, unauthenticated data. It also refactors the database seeding service to be non-destructive, improving the integrity of initial data.

🔓 Public Access for Generic Data Route

The generic /api/v1/data route has been re-architected to support both public (unauthenticated) and private (authenticated) access. This was necessary to expose non-sensitive data, such as the RemoteConfig, to clients before they log in, without creating separate, dedicated public endpoints.

• A requiresAuthentication flag has been added to the model registry, allowing authentication to be configured on a per-model, per-action basis. [#57]
• A new _conditionalAuthenticationMiddleware was implemented to dynamically apply authentication based on this configuration. [#57]
• The RemoteConfig model's GET action has been configured with requiresAuthentication: false, making it the first publicly accessible model via this route. [#57]
• Core middleware (authorization, ownershipCheck) and route handlers were refactored to correctly handle a nullable User object, ensuring proper behavior for anonymous requests. [#57]
• The data route's rate limiter was updated to use the client's IP address for unauthenticated requests. [#57]

🗄️ Robust & Non-Destructive Database Seeding

The database seeding service was refactored to ensure initial data population is more reliable and does not accidentally overwrite existing data.

• The generic _seedCollection method was updated to use the MongoDB $setOnInsert operator. This ensures that seeding operations will only create new documents and will never overwrite existing ones. [#56]
• The logic for seeding the initial RemoteConfig document now explicitly checks if a document already exists, preventing accidental overwrites of a live configuration. [#56]
• The initial seed for RemoteConfig now sets the primary ad platform to AdPlatformType.local by default, ensuring a production-ready setup. [#56]

08-09-2025

📢 New Entity Support: Local Ads

The API required a mechanism to store, manage, and control access to locally-defined ad content, distinct from external ad network integrations. This work integrates the new LocalAd data model into all relevant backend systems, including permissions, the generic data access layer, and database management.

• Integrated the new LocalAd data model and its repository into the application's core dependency injection system. [#55]
• Extended the Role-Based Access Control (RBAC) system with specific CRUD permissions, granting read access to guest users and full control to administrators. [#55]
• Registered LocalAd in the polymorphic data operation registry, enabling it to be managed via the generic /data API endpoints. [#55]
• Added support for database seeding and created a new index on the adType field for efficient querying of the LocalAd collection. [#55]

25-08-2025

🚀 A New, Specialized Country Query Service

To support more advanced filtering needs that go beyond simple data lookups, we've implemented a dedicated service that leverages the power of MongoDB's aggregation framework. This provides a robust and efficient solution for complex data discovery.

• 🎯 Introduced CountryQueryService: A new service was launched to handle complex queries, such as finding countries with active sources or headlines. It features an in-memory cache with a 15-minute TTL to dramatically boost performance for frequent requests. #53
• 🔧 Refined Aggregation Logic: The service's MongoDB aggregation pipeline was improved to provide more accurate results by switching to a case-insensitive regex search and correcting Object ID comparisons in data lookups. #54

⚡️ API Refinements & Database Performance

Alongside the new service, we made key optimizations to the database and API to ensure these new capabilities are fast and easy to use.

• 🔎 Implemented Country Name Filtering: The API now supports searching for countries by name. This is backed by a new text index on the countries collection to ensure these search queries are highly performant. #50
• ⚙️ Optimized Query Parameters & Indexing: The country search parameter was updated from name to the more conventional q. Additionally, a new database index was added to significantly speed up queries that filter headlines by the event country's name. #51

12-08-2025

🚀 Major Refactor for Maintainability & Robustness

This release marks a significant architectural overhaul of the API's core data routes. The primary focus was not on adding new features, but on a deep refactoring effort to improve maintainability, extensibility, and resilience. We've centralized logic, streamlined middleware, and enhanced data model consistency.

---

🏛️ Architectural Refactor: Middleware & Data Operations

We've fundamentally changed how the API handles data requests to make the system cleaner and much easier to extend in the future. The big-picture goal was to move logic out of individual route handlers and into single-responsibility components.

Our Rationale: Large switch statements and duplicated logic in every route handler are hard to maintain. By centralizing data operations and middleware responsibilities, we can add new data models or change behavior in one place, rather than many.

• New DataOperationRegistry: This is the centerpiece of the refactor. A new registry now centrally manages all CRUD (Create, Read, Update, Delete) operations, completely eliminating the large, cumbersome switch statements from our route handlers.
• Smarter Middleware Chain:
  • A new DataFetchMiddleware is now solely responsible for fetching items by ID, making the data available to other middleware in the request context.
  • The OwnershipCheckMiddleware has been simplified. It no longer fetches data itself; it now just validates ownership on the pre-fetched item.
• Optimized Database Calls: By passing fetched data through the request context, we've eliminated redundant database reads that were occurring in the old workflow.

🛡️ Enhanced Error Handling & Resilience

We've hardened the API to be more resilient against malformed requests, ensuring it provides clear and actionable feedback instead of generic server errors.

• Robust Query Parameter Parsing: Added try-catch blocks around the parsing of filter and sort query parameters. Invalid JSON or incorrect formats now return a specific BadRequestException.
• Safe Request Body Handling: The logic that converts incoming JSON to data models in POST requests is now wrapped in a try-catch block, gracefully handling malformed bodies.
• Improved Logging: Implemented more comprehensive logging throughout the data routes, including error and stack trace details, to improve visibility for debugging.

🔄 Data Model & Synchronization Updates

To keep the API in sync with changes across the entire toolkit, we've updated several data models and permissions.

• Data Model Consistency:
  • The feedActionStatus field has been renamed to feedDecoratorStatus to align with terminology used in the client applications.
  • A user's language preference is no longer a simple string ('en') but a structured Language object, improving data integrity.
• Secure Guest User Updates: Guest users are now permitted to update a secure, whitelisted portion of their own user object (specifically, their feedDecoratorStatus), which is essential for certain client-side features. This is handled with a secure partial update that prevents any changes to sensitive fields.

03-08-2025

🚀 Enhanced Security, Admin Control & Data Refactoring

This release brings foundational improvements to the API, focusing on security, system stability, and a significant refactoring of how data is seeded and managed. These changes make the API more robust, secure, and easier to manage from an administrative perspective.

---

🛡️ Security & Stability Enhancements

We've introduced two major features to protect the API from abuse and ensure stable operation.

• API Rate Limiting: A new, configurable rate-limiting system has been implemented to prevent abuse of key endpoints.

  • It is applied to the authentication endpoint (/auth/request-code) based on IP address and to the data API (/api/v1/data) based on the authenticated user's ID.
  • Limits and time windows are easily configurable via new environment variables.
  • A rate_limiting.bypass permission is granted to Admin and Publisher roles, ensuring administrative tasks are not affected.
  • The system now correctly returns an HTTP 429 (Too Many Requests) status code when a limit is exceeded.

• Declarative Admin Account Management: We've introduced a more robust way to manage the system's administrator account.

  • A new OVERRIDE_ADMIN_EMAIL environment variable allows you to securely define the sole administrator.
  • On startup, the database seeding process will automatically create or update the admin user based on this variable, simplifying initial setup and account recovery.

💾 Database Seeding & Data Model Refactor

The database seeding process has been significantly refactored. The goal was to separate foundational system data from dynamic content fixtures, making the initial setup cleaner and more predictable.

Our Rationale: The core database seed should only be responsible for setting up the essential, static data required for the application to function—not for populating it with sample content like headlines or topics.

• Leaner Seeding Process: The default seeding operation no longer populates the database with content fixtures. Its primary responsibilities are now reduced to creating database indexes and seeding the override admin user.
• New Language Entity: A new Language data model has been introduced as a first-class entity within the API, complete with its own repository and role-based access permissions.
• Read-Only Static Data: The Country and newly added Language models are now configured as read-only through the API. This enforces the new philosophy that this type of foundational data is managed via the database seeding process, not through user-driven API calls.

23-07-2025

🚀 Project-Wide Renaming & Standardization

This release marks a significant foundational refactoring focused on renaming the core project and all internal dependencies. The primary goal is to align with the new 'Flutter News App Toolkit' ecosystem and establish a more conventional, maintainable, and descriptive codebase.

This is a breaking, comprehensive change that updates nearly every file to use the new naming conventions, from package imports to type definitions.

✨ Renaming & Rebranding Initiative

• Project Rebranded: The core project has been officially rebranded from 'Headlines Toolkit' to 'Flutter News App Toolkit'. The API repository itself is now named flutter_news_app_api_server_full_source_code.
• Internal Packages Renamed: All internal private dependencies, previously prefixed with ht_ (e.g., ht_data_client, ht_shared), have been renamed to remove the prefix (e.g., data_client, core).
• Documentation & Metadata Updated: The README.md and pubspec.yaml have been updated to reflect the new project name, description, and repository URLs.

🔧 Codebase & Standardization

• Codebase Adapted: All import statements and type annotations have been updated across the entire codebase to use the new package names (e.g., HtDataRepository is now DataRepository).
• Standardized Error Handling: The custom HtHttpException has been replaced with the more standard HttpException type for consistent error handling across the application.

🔧 Configuration & Tooling Updates

• Configuration Aligned: The .env.example file has been updated with the new database name to match the new branding.
• Dependabot Updated: The dependabot.yaml configuration has been updated to correctly ignore the newly named private dependencies.
• Dev Dependency Upgrade: The very_good_analysis dev dependency has been upgraded to its latest version.

21-07-2025

✨ SendGrid Integration & Configuration Refinements

This release marks a significant upgrade to our email capabilities by migrating from an in-memory solution to SendGrid for sending real transactional emails. To support this major feature, we have also refactored our application configuration and environment variable handling to be more robust, maintainable, and secure.

✨ Transactional Email via SendGrid

• Email Service Migration: The API's email service has been migrated to use SendGrid, enabling the application to send real transactional emails, such as One-Time Passwords (OTPs), for the first time.
• New Environment Configuration: New environment variables (SENDGRID_API_KEY, DEFAULT_SENDER_EMAIL, OTP_TEMPLATE_ID) have been introduced to configure the SendGrid integration. The .env.example and documentation have been updated accordingly.
• Auth Service Integration: The AuthService has been updated to use the new email service, passing the configured sender email and template ID for sending OTPs.

🔧 Configuration & Code Quality Improvements

• Robust Environment Variable Handling: A new helper method now centralizes the logic for retrieving required environment variables. This reduces boilerplate code and ensures the application fails fast if a critical variable is missing.
• Refined API Client Setup: The SendGrid API client initialization has been improved by separating the base URL from the version path, making the configuration clearer and more flexible.
• Code Cleanup: General code cleanup was performed across several files, including removing obsolete in-memory email code and applying formatting adjustments for better readability.

20-07-2025

🚀 Authentication Overhaul & Production-Ready Enhancements

This release delivers a comprehensive overhaul of the API's authentication system and backend infrastructure. The primary focus has been on creating a seamless and secure experience for guest users converting to permanent accounts.

Simultaneously, we've executed a major push to harden the entire platform for production by migrating critical authentication data to a persistent store, externalizing sensitive configurations, and implementing performance-boosting database indexes.

✨ Seamless & Secure Guest User Conversion

The entire lifecycle for guest users has been re-architected to be more intelligent, secure, and streamlined.

• Intelligent Sign-In Flow: The authentication process now smartly handles guest users. If a guest signs in with a new email, their account is seamlessly converted to a permanent one. If they use an email that already exists, they are logged into the existing permanent account, and their old guest data is securely cleaned up.
• Simplified API: The dedicated /link-email endpoints have been removed. All guest conversion logic is now consolidated directly into the main sign-in flow, creating a simpler and more robust API surface.
• Automated Token Invalidation & Cleanup: When a guest converts or signs into an existing account, their old guest token is automatically invalidated and their temporary guest account is deleted, preventing orphaned data and improving security.
• Reliable User Data Provisioning: A centralized helper now ensures that all necessary user documents (like settings and preferences) are idempotently created for any new user, whether they register directly or are converted from a guest account.

🛡️ Production-Ready Security & Persistence

We've moved beyond in-memory solutions and hardcoded secrets to a more robust and secure architecture.

• Persistent Authentication Data: Token blacklists and active verification codes are no longer stored in-memory. They are now persisted in dedicated MongoDB collections, ensuring they survive server restarts.
• Automated Data Cleanup with TTL Indexes: We've implemented MongoDB's Time-To-Live (TTL) indexes on the new authentication collections. This automates the cleanup of expired tokens and verification codes directly within the database, ensuring excellent data hygiene with no manual intervention.
• Externalized JWT Configuration: All sensitive JWT secrets, issuer URLs, and token expiry settings are now loaded from environment variables instead of being hardcoded, following security best practices.
• Structured Logging: All print() statements have been replaced with a structured Logger, providing vastly improved observability for monitoring and debugging.

🚀 Performance & Backend Modernization

• Automated Text Search Indexing: The database seeding process now automatically creates text search indexes on headlines, topics, and sources. This significantly improves the performance and efficiency of search operations.
• Migration to MongoDB ObjectIDs: The application now uses MongoDB's native ObjectId for all unique identifier generation, replacing the generic uuid package. This aligns our ID strategy with our database technology.
• Standardized Timestamps: The API now automatically and consistently sets createdAt and updatedAt fields for all new and updated data, ensuring data integrity.

14-07-2025

🛡️ API Response & Error Handling Overhaul

This release delivers a comprehensive overhaul of our API's response handling, ensuring that client applications receive data and errors in a consistent, predictable, and secure manner. We have standardized the format for both successful responses and critical error messages, while also fixing a significant CORS issue that impacted error visibility.

✨ Key Enhancements

Critical Error Handling & CORS Fix

• Fixed Unreadable Error Messages: Resolved a critical issue where browser-based clients could not read API error messages due to CORS policies. Error responses now dynamically include the necessary Access-Control-Allow-Origin headers, ensuring clients can always access detailed error information.
• Centralized Error Generation: All JSON error responses are now generated by a single helper function, standardizing their format and ensuring consistent behavior across the API.
• New CORS Configuration: A new CORS_ALLOWED_ORIGIN environment variable has been introduced, allowing for explicit configuration of the allowed origin in production environments.

Standardized Success Responses

• New Response Helper: A ResponseHelper utility has been introduced to standardize the format of all successful API responses. This reduces boilerplate code in route handlers and ensures a consistent SuccessApiResponse structure across all endpoints.
• Widespread Refactoring: Numerous route handlers have been refactored to use the new helper, significantly cleaning up the codebase and improving maintainability.

🔧 Code Quality & Organization

• Improved Code Organization: The RequestId class has been moved to its own dedicated file for better modularity.
• Logging Cleanup: Redundant request ID prefixes have been removed from log statements, as this is now handled more cleanly through the request context.