feat!: v6.0.0 - schema normalization, whitelist mode, and bug fixes #47
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BREAKING CHANGE: Media metadata is now stored exclusively in the media table. The media_type, media_id, and media_path columns have been removed from the messages table. Changes: - Remove media_type, media_id, media_path from messages table - Add proper FK constraints (sender_id->users, media->messages, reactions->users) - Add performance indexes for reply lookups, media filtering, username searches - Update API to return media as nested object instead of flat fields - Update frontend to use msg.media?.type syntax - Service messages now use raw_data.service_type - Polls now detected by raw_data.poll presence Migration: - Alembic migration 005 handles data migration automatically - Creates backup table for rollback capability - Supports both SQLite and PostgreSQL
|
🐳 Dev images published!
The dev/test instance will pick up these changes automatically (Portainer GitOps). To test locally: docker pull drumsergio/telegram-archive:dev
docker pull drumsergio/telegram-archive-viewer:dev |
sender_id in messages can contain channel/group IDs (negative values) that don't exist in the users table. Removed the FK constraint but kept the ORM relationship for joins.
|
🐳 Dev images published!
The dev/test instance will pick up these changes automatically (Portainer GitOps). To test locally: docker pull drumsergio/telegram-archive:dev
docker pull drumsergio/telegram-archive-viewer:dev |
|
🐳 Dev images published!
The dev/test instance will pick up these changes automatically (Portainer GitOps). To test locally: docker pull drumsergio/telegram-archive:dev
docker pull drumsergio/telegram-archive-viewer:dev |
1 similar comment
|
🐳 Dev images published!
The dev/test instance will pick up these changes automatically (Portainer GitOps). To test locally: docker pull drumsergio/telegram-archive:dev
docker pull drumsergio/telegram-archive-viewer:dev |
|
🐳 Dev images published!
The dev/test instance will pick up these changes automatically (Portainer GitOps). To test locally: docker pull drumsergio/telegram-archive:dev
docker pull drumsergio/telegram-archive-viewer:dev |
|
🐳 Dev images published!
The dev/test instance will pick up these changes automatically (Portainer GitOps). To test locally: docker pull drumsergio/telegram-archive:dev
docker pull drumsergio/telegram-archive-viewer:dev |
|
🐳 Dev images published!
The dev/test instance will pick up these changes automatically (Portainer GitOps). To test locally: docker pull drumsergio/telegram-archive:dev
docker pull drumsergio/telegram-archive-viewer:dev |
|
🐳 Dev images published!
The dev/test instance will pick up these changes automatically (Portainer GitOps). To test locally: docker pull drumsergio/telegram-archive:dev
docker pull drumsergio/telegram-archive-viewer:dev |
Albums are now handled entirely via grouped_id in the NewMessage handler: - Added grouped_id capture to listener's NewMessage handler - Removed separate Album event handler (was causing duplicate processing) - Removed LISTEN_ALBUMS config option The viewer already groups messages by grouped_id, so albums display correctly without a separate handler. This simplifies the codebase and fixes issue #46.
|
🐳 Dev images published!
The dev/test instance will pick up these changes automatically (Portainer GitOps). To test locally: docker pull drumsergio/telegram-archive:dev
docker pull drumsergio/telegram-archive-viewer:dev |
The docker-compose.yml was using `${CHAT_TYPES:-default}` syntax which
treats empty values the same as unset, always using the default.
Changed to `${CHAT_TYPES-default}` (without colon) which only uses the
default when the variable is completely unset, allowing CHAT_TYPES=
(empty) to work for whitelist-only mode as documented.
Also updated README with a note explaining this syntax difference for
users who may have older docker-compose.yml files.
|
🐳 Dev images published!
The dev/test instance will pick up these changes automatically (Portainer GitOps). To test locally: docker pull drumsergio/telegram-archive:dev
docker pull drumsergio/telegram-archive-viewer:dev |
## Problem Users expected `CHANNELS_INCLUDE_CHAT_IDS` to work as a whitelist (backup ONLY these chats), but it was actually additive (ALSO backup these chats in addition to what CHAT_TYPES selects). This caused confusion and made it difficult to backup only specific chats. ## Solution Added a new `CHAT_IDS` environment variable that provides true whitelist mode: ```bash # Backup ONLY these 2 chats - nothing else CHAT_IDS=-1001234567890,-1009876543210 ``` When `CHAT_IDS` is set, ALL other filtering options are ignored. ## Two Filtering Modes | Mode | When | How it works | |------|------|--------------| | Whitelist | CHAT_IDS set | Backup ONLY listed chats | | Type-based | CHAT_IDS not set | Use CHAT_TYPES + INCLUDE/EXCLUDE | ## Files Changed - src/config.py: Parse CHAT_IDS, add whitelist_mode flag - src/listener.py: Respect whitelist mode in _should_process_chat - docker-compose.yml: Add CHAT_IDS variable with documentation - README.md: Rewrite chat filtering section with clear examples - .env.example: Update with new CHAT_IDS option - docs/CHANGELOG.md: Document the feature Closes #48
GeiserX
changed the title
|
🐳 Dev images published!
The dev/test instance will pick up these changes automatically (Portainer GitOps). To test locally: docker pull drumsergio/telegram-archive:dev
docker pull drumsergio/telegram-archive-viewer:dev |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Major release with schema normalization, new whitelist filtering mode, and several bug fixes.
Issues Addressed
grouped_idCHAT_IDSvariable for simple whitelist filteringBreaking Changes
Schema Normalization (addresses #45)
Media metadata moved from
messagestable to dedicatedmediatable:Removed columns from
messages:media_typemedia_idmedia_pathAPI response format changed:
Removed
LISTEN_ALBUMS(fixes #46)Albums are now automatically handled via
grouped_idin the NewMessage handler. The separate Album event handler was redundant and has been removed.New Features
CHAT_IDSWhitelist Mode (fixes #48)New simple way to backup only specific chats:
# Backup ONLY these 2 chats - nothing else CHAT_IDS=-1001234567890,-1009876543210When
CHAT_IDSis set, ALL other filtering options are ignored.CHAT_IDS=id1,id2CHAT_TYPES=channelsThis addresses the common confusion where users expected
*_INCLUDE_CHAT_IDSto act as a whitelist.Database Changes
Foreign Keys Added
media(message_id, chat_id)→messages(id, chat_id)with CASCADE deletereactions.user_id→users.idwith SET NULL on deleteNew Indexes
idx_messages_reply_to- Fast reply lookupsidx_media_downloaded- Find undownloaded mediaidx_media_type- Filter by media typeidx_reactions_user- User reaction queriesidx_chats_username/idx_users_username- Username lookupsMigration
Automatic via Alembic. The migration:
Backup your database before upgrading.
Test Plan
grouped_idCHAT_IDSwhitelist mode works