Skip to content

1.8.7#174

Open
awehttam wants to merge 172 commits intomainfrom
claudesbbs
Open

1.8.7#174
awehttam wants to merge 172 commits intomainfrom
claudesbbs

Conversation

@awehttam
Copy link
Owner

@awehttam awehttam commented Mar 15, 2026

Summary by CodeRabbit

  • New Features

    • Advanced per-field/date message search, print-friendly message view, and admin message edit
    • FREQ (file request) support with logging, outbound queue, magic-name listings, and Gemini file-area browsing/server
    • Nodelist map with geocoding and map-data API; new Database Statistics and FREQ Log admin pages
  • Improvements

    • PETSCII/C64 badges and enhanced art-format detection
    • Configurable file upload/download credit costs/rewards
    • Multiple database indexing and query-performance optimizations
  • Documentation

    • Upgrade guide, FREQ, geocoding, credit system, and related proposals/docs added or expanded

awehttam and others added 30 commits March 13, 2026 20:41
Add art_format to echomail list queries so the JS can detect PETSCII
messages, and display a small C64 badge beside the envelope icon in
both the echomail and netmail message lists.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The C64 PETSCII badge was only visible from the two non-threaded flat
list queries. All remaining list queries (threaded view, subscribed
areas, thread context loading, child/parent loading) were also missing
art_format from their SELECT clause.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sysops can now edit art_format and message_charset on any echomail
message directly from the message reader. Netmail senders/receivers
can do the same for their own messages. Edit button (pencil icon)
lives in the message header toolbar alongside prev/next navigation.

Also adds a C64 badge in message lists for PETSCII-detected messages,
fixes art_format missing from echomail list queries in MessageHandler,
and updates i18n keys, API routes, and UPGRADING_1.8.7 docs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Advanced Search modal on echomail and netmail with per-field filters
  (poster name, subject, body) and date range picker; fields ANDed together
- Simple search bar unchanged
- Backend: field-specific ILIKE conditions built separately from general q=
  search; date range computed in PHP to avoid PDO/pgsql ::cast issues
- Performance: derive echoarea and filter counts from already-fetched results
  instead of running two additional full-table ILIKE scans
- Add trigram GIN indexes (pg_trgm) on echomail/netmail subject and
  message_text for fast substring search
- Add index on echomail(date_received) for date range filtering
- Fix search returning blank response caused by bytea raw_message_bytes column
  being included in SELECT em.* and returned as unserializable PHP resource
- Fix stale message list showing after search failure (fail handler now clears
  the container)
- Update UPGRADING_1.8.7.md with search and reindex notes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New /admin/database-stats page with six tabs: size & growth, activity,
  query performance, replication, maintenance health, and index health
- DatabaseStats class queries pg_stat_* views; gracefully degrades when
  extensions (pg_stat_statements) are absent
- Dashboard now shows database size with link to stats page
- Optional daemons (telnet, SSH, Gemini, MRC, multiplexer) shown in
  service status with Running/Stopped badges; no PID = stopped
- database_maintenance.php now vacuums all user tables (via
  pg_stat_user_tables) and prints each table as it's processed
- Migration v1.11.0.13 drops 10 redundant explicit indexes that duplicate
  unique-constraint indexes on the same columns
- Proposal doc for redundant index drops with verification notes
- CLAUDE.md note: do not create explicit indexes on UNIQUE columns
- UPGRADING_1.8.7.md updated with database stats section; removed
  incorrect "no schema migration" note

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New tab on /admin/database-stats showing per-locale, per-file key
  counts, file sizes, and serialized memory footprint of i18n catalogs
- Summary cards give a quick overview per locale; detail table groups
  files under each locale with totals row
- Fix back button using correct ui.common.go_back key
- i18n keys added for en, es, fr

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fills gaps in fr/common.php relative to en/common.php:
- ui.common.advanced_search.* (advanced search modal)
- ui.common.db_id, ui.common.message_id
- ui.base.admin.bbs_directory, echomail_robots, bbs_lists
- ui.admin.bbs_directory.* (full admin BBS directory UI)
- ui.admin.echomail_robots.*
- ui.admin.bbs_settings.features.enable_bbs_directory / bbs_directory_help
- ui.bbs_directory.* (public directory page)
- ui.echomail.art_format*, edit_message*, message_charset*
- ui.files.edit, edit_file, move_to_area, scan_status_*, short_description,
  virus_name_placeholder

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bootstrap's contextual table classes use bright solid backgrounds that make
text unreadable on dark themes. Override them in cyberpunk, dark, greenterm,
and amber to use low-opacity tinted backgrounds that preserve the semantic
colour signal without drowning out foreground text.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous fix only set --bs-table-bg but not --bs-table-striped-bg, so
table-warning rows alternated between the tinted dark colour (odd/striped
rows) and Bootstrap's bright #fff3cd (even rows). Add all Bootstrap table
CSS variable overrides (striped, active, hover variants) so every row in a
contextual table class renders consistently dark across all four themes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The daemon polls every 100ms accumulating millions of seq scans. The old
index (sent_at, priority) didn't match ORDER BY priority DESC, created_at ASC
so PostgreSQL chose seq scans. The new partial index covers only unsent rows
with the exact sort order, enabling efficient index scans.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…sages

users: add UNIQUE LOWER(username) index — covers login queries and OR-based
name-matching in mail delivery that were forcing seq scans on every auth request.

shared_messages: replace (message_id, message_type) index with composite
(message_id, message_type, shared_by_user_id) WHERE is_active = TRUE to cover
the LEFT JOIN on every echomail listing page load.

saved_messages: replace (message_id, message_type) index with composite
(message_id, message_type, user_id) matching the echomail-driven LEFT JOIN.

chat_messages: replace total-count polling with incremental max-ID approach.
The old query counted ALL messages on every 30s poll (full table scan with OR
across three columns). New approach queries only rows newer than the last seen
message ID, using the PK index. Stores last_chat_max_id in user meta;
JS passes chat_max_id from stats response to markSeen.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…er script

Map features:
- Nodes grouped by system_name into a single marker per BBS; popup lists all
  network addresses the system appears in across different nodelists
- Colour-coded markers by zone (blue=Z1, green=Z2, amber=Z3, red=Z4,
  purple=Z5, teal=Z6, gold=multi-zone, grey=unknown)
- Map legend in bottom-right corner
- Lazy-loads via /api/nodelist/map-data on first tab click
- Leaflet MarkerCluster for dense areas

Geocoder:
- scripts/geocode_nodelist.php: geocode nodelist location strings using the
  shared bbs_directory_geocode_cache; supports --limit, --force, --dry-run
- Migration v1.11.0.18: latitude/longitude columns on nodelist table
- BbsDirectoryGeocoder: removed 32-day TTL; cache is now permanent

To populate coordinates after upgrading:
  php scripts/setup.php
  php scripts/geocode_nodelist.php

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Condense stats alert to a single line with middle-dot separators;
  Last imported badges moved to a second line beneath the counts
- Add Nodelist Map section to UPGRADING_1.8.7.md covering geocoding
  script usage, cron setup, zone colour coding, and new migration

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Copy dark CSS treatment from bbs_directory (dark bg, tile filters,
  dark controls and popups) to the nodelist Leaflet map

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documents both geocoder scripts (nodelist and BBS directory),
shared cache behaviour, Nominatim rate limiting, environment
variables, and cron usage.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename bbs_directory_geocode_cache → geocode_cache (migration v1.11.0.19)
  so it serves as a common geocoding service for both BBS Directory and Nodelist
- Rename backfill_bbs_directory_geocoding.php → geocode_bbs_directory.php
  to match the geocode_nodelist.php naming pattern
- Update all references in BbsDirectoryGeocoder, scripts, and docs

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Colours are now assigned dynamically from a palette as domains are
encountered in the data. Nodes on multiple domains use gold. The
legend is built at runtime showing actual domain names.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…onse

The networks array was missing the domain field so all markers fell
through to the unknown (grey) colour. Add n.domain to the SELECT and
pass it through in each network entry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add fa-code icon button beside the edit button in echomail and
  netmail modal headers; button gains 'active' class when open
- Remove the kludge lines heading/button from inside the message body
- Kludge container stays hidden by default; toggleKludgeLines() updated
  to use button active state instead of updating text/icon inside body
- Document change in UPGRADING_1.8.7.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Print button (fa-print) opens message in a clean popup window;
  window auto-closes via onafterprint after the dialog completes
- printMessage() defined in echomail.js and netmail.js directly to
  avoid service worker cache timing issues
- Kludge toggle moved to fa-code icon button in modal header; container
  hidden by default, button shows active state when open
- Add ui.common.print i18n key (en + es)
- Document in UPGRADING_1.8.7.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
awehttam and others added 30 commits March 17, 2026 16:14
…ho areas

- New script scripts/send_echomail_digest.php — run hourly via cron; enforces
  per-user daily/weekly frequency internally; license-gated; supports --dry-run,
  --verbose, --user=ID flags
- Migration v1.11.0.31: adds echomail_digest (none/daily/weekly) and
  echomail_digest_last_sent to user_settings
- Settings UI: frequency selector in Notifications section; locked with
  'Registered Feature' badge when unlicensed
- MessageHandler: DIGEST_FREQUENCY type validation for the new setting
- i18n keys added to en and es catalogs
- PremiumFeatures.md: moved echomail digest from future ideas to implemented table
- UPGRADING_1.8.7.md: documented cron setup and --dry-run test usage

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds /admin/register route that renders REGISTER.md via MarkdownRenderer,
reusing the upgrade_notes template. The menu item appears in the Help submenu
only when license_valid is false, separated by a divider and styled in blue
to draw attention.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Daily digest looks back 24 hours, weekly looks back 7 days — both on first
run and subsequent runs. Eliminates the catch-up problem without a separate
first-run special case.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Version table: expand 1.8.7 highlights
- Web Interface: add global file search, file preview/ISO areas, outbound FREQ,
  artwork encoding editor, email notifications, registration blurb
- File Areas section: subfolder nav, file preview (corrected D64 = PRG gallery),
  ISO-backed areas, global file search
- Scripts table: add send_echomail_digest.php
- Cron section: add echomail digest hourly entry

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Digest links used echoarea numeric ID in URL; route expects the tag name.
  Changed to /echomail/<tag> in both plain text and HTML builders.
- PHPMailer defaults to ISO-8859-1, causing em dashes and other non-ASCII
  chars to render as junk (â€") in subjects and HTML bodies. Set CharSet
  to UTF-8 on all PHPMailer instances in Mail.php.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…hout clearing state

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…age title

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ystem name placeholder

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Change .dropdown-item-locked from pointer-events:none to pointer-events:auto
  so native title tooltips are visible on hover; use cursor:not-allowed
- Add onclick="return false;" to locked nav links to prevent navigation
- Dashboard: show Economy Viewer and Referral Analytics buttons when unlicensed,
  styled as disabled with lock icon and Registered Feature title tag
- Fix UPGRADING_1.8.7 Economy Viewer URL and unlicensed behavior description
- Bump SW cache to v325

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PHP-served HTML bypasses the SW cache, so this acts as a hard refresh on
every navigation — ensuring mobile users pick up SW updates without needing
to manually clear the cache.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New docs/proposals/FileAreaComments.md: full design for file area
  echomail comments — kludge + subject matching, threaded display capped
  at 3 levels, modal UI with blurred login overlay for guests, create-or-
  select linked echo area in file area admin
- PremiumFeatures: move file area comments to community edition; reference
  new proposal doc

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…back

- File area comments: echomail-based comment threads on file entries, linked
  via ^AFILEREF kludge; comments shown in both details and preview modals;
  FILEREF kludge banner displayed in echomail reader linking back to the file
- Comment echo area admin UI redesigned: single select + collapsible new-tag
  input, integrated into main Save button flow; auto-selects LVLY_FILECHAT
  for lovlynet file areas if it exists
- ZIP browser: replaced FILE_ID.DIZ-only preview with a full browsable file
  listing; per-entry preview for images, video, audio, text, ANSI, RIP,
  PETSCII; new /zip-contents and /zip-entry API endpoints
- RIPscrip preview: .rip files previewed server-side via RipScriptRenderer;
  RIP gallery for multiple .rip files inside a ZIP
- Legacy ZIP fallback: entries using old DOS compression methods (implode,
  shrink) now attempted via system unzip; graceful 415 error with download
  link if unavailable; legacy badge shown in ZIP browser listing
- ZIP entry comp_method included in /zip-contents response
- i18n keys added for all new UI strings (en + es)
- Migration v1.11.0.32 for file_area_comments table

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- LovlyNetClient: switch from file_get_contents to cURL; send both
  Authorization: Bearer and X-API-Key headers for compatibility
- Admin page /admin/lovlynet: tabbed echo/file area list with
  subscribe/unsubscribe toggles; reads credentials from lovlynet.json
- Proxy routes: GET /admin/api/lovlynet/areas,
  POST /admin/api/lovlynet/subscription
- Nav link added under Area Management in both base templates
- i18n keys added (en + es); heading updated to "LovlyNet Subscriptions"
- UPGRADING_1.8.7.md: added ZIP browser, RIPscrip preview, and
  LovlyNet Subscriptions sections; updated TOC and summary
- 1.8.7post.txt: echomail announcement post for 1.8.7 preview

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ents

- Add prev/next arrow navigation in ZIP entry viewer with N/total counter
- Show download button on ALL ZIP entries (not just non-previewable)
- Show legacy compression badge for unsupported methods (implode/shrink)
- Add shell fallback chain (unzip/unzip.exe/7z/7za) for legacy extraction
- Add Windows NUL support and unzip.exe detection in zip-diag endpoint
- Fix modal-dialog-scrollable so comments section is visible on all previews
  (ZIP, MP4, and any other preview type that uses the shared preview modal)
- Remove max-height:78vh from ZIP browser inner div (modal handles scroll)
- Bump SW cache to v338

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…AFILEREF

- Use recursive CTE to fetch entire thread by reply_to_id from root(s) matched
  by FILEREF kludge or subject — replies without the kludge are now included
- AFILEREF kludge now written as AREATAG@domain per LSC2 spec
- Both GET and POST comment endpoints match new (with domain) and legacy
  (without domain) kludge patterns for backward compatibility
- JS banner link strips @Domain before passing tag to /files?area= filter
- Bump SW cache to v339

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a file's extension is not in any known list, sample the first 4 KB
and serve as text/plain if: no null bytes found and ≥90% of bytes are
printable (ASCII printables, tab/CR/LF, or high bytes for UTF-8/CP437).

- /api/files/{id}/preview: streams UTF-8 files via readfile(); CP437
  files ≤1 MB get iconv conversion; larger non-UTF-8 served raw
- /api/files/{id}/zip-entry: same heuristic on already-in-memory content
- JS renderPreviewContent: probes preview URL, renders if text/plain
- JS renderZipEntry: probes zip-entry URL, checks Content-Type before
  rendering to avoid displaying binary garbage as text

Covers files like MICRONET.030 (FidoNet nodelist) and similar BBS-era
files with numeric or non-standard extensions.

Bump SW cache to v341.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- .md files now render as HTML via marked.js v13 instead of plain text
- Markdown rendering works in both the file preview modal and ZIP entry viewer
- All text <pre> previews get text-align:left to override Bootstrap text-center
- Bundle marked.min.js in public_html/vendor/marked-13/
- Load marked.js in files.twig and shared_file.twig before file-preview.js
- Falls back to plain text <pre> if marked is not available
- Bump SW cache to v342

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Render Markdown server-side using the existing MarkdownRenderer class
rather than bundling a third-party JS library. Both /api/files/{id}/preview
and /api/files/{id}/zip-entry now return text/html for .md files;
the client injects the HTML fragment directly.

Removes marked.min.js vendor bundle added in previous commit.
Bump SW cache to v343.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tching

The FILEREF kludge was being added via a post-INSERT UPDATE, after
spoolOutboundEchomail() had already read the row — so remote nodes never
received the kludge. Fix: pass it as prependKludges to postEchomail() so
it is in the INSERT and included in the spool.

Also improve comment thread matching:
- Case-insensitive subject match (LOWER()) handles filename case differences
  between servers
- Kludge-based anchor no longer restricted to reply_to_id IS NULL (FTN
  reply-matching on receiving side may set reply_to_id on thread roots)
- DISTINCT to deduplicate when both kludge and subject match same message
- Add $prependKludges parameter to MessageHandler::postEchomail()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Lazily sync files.comment_count when GET /files/{id}/comments runs,
  so badges stay accurate for comments received via FTN (which bypass
  the normal post path)
- Backfill comment_count for all files in an area when comment_echoarea_id
  is linked, so badges appear immediately without requiring each file
  to be individually viewed
- Fix fileareas.twig auto-selecting LVLY_FILECHAT in the comment echo
  area dropdown when comment_echoarea_id is actually NULL, which made
  the link appear configured when it was not saved

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switch banner from alert-secondary (light gray) to alert-info (teal/blue)
so it reads clearly against dark themes. Bump SW cache to v344.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove webaudio-mod-player integration from file-preview.js (ScriptProcessorNode
  and BiquadFilter instability made audio unworkable)
- Fix TicFileGenerator to write 'Pw -' instead of 'Pw ' when no password is
  configured; empty Pw field caused remote nodes to reject with "Password missing"
- Bump service worker cache to binkcache-v348 to force fresh JS delivery

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add network filter dropdown to /echolist using the same checkbox-dropdown
  pattern as the nodelist flag filter; supports Local and all FTN domains
- Add i18n key ui.echolist.all_networks (en + es)
- Bump SW cache to binkcache-v349
- Document Echo List Network Filter and TIC password field bug fix in UPGRADING_1.8.7

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Individual file areas can now be flagged as public, allowing
unauthenticated visitors to browse and download files without a BBS
account. Includes an optional /public-files index page controlled by a
new BBS setting.

- Migration v1.11.0.33_public_file_areas.sql adds is_public column
- FileAreaManager: canAccessFileArea() allows null userId for public areas;
  updateFileArea() saves is_public (license-gated)
- Web route /files/{tag}: skips requireAuth() for public areas
- New web route /public-files: guest-accessible area index
- API: GET /api/files, /api/files/{id}, /api/files/{id}/preview,
  /api/files/{id}/download, /api/files/{id}/zip-contents,
  /api/files/{id}/zip-entry all support guest access on public areas
- fileareas.twig: Public checkbox (license-gated)
- files.twig: sidebar hidden for guests; loadFileComments skipped for
  guests to prevent 401 triggering global login redirect; OG meta tags
- public_files.twig: new area index template
- base.twig + shells/web/base.twig: Public Files nav link for guests
- Admin BBS settings: Enable Public Files Index toggle (license-gated)
- i18n: en/es keys for all new UI strings
- Docs: FileAreas.md, UPGRADING_1.8.7.md, PremiumFeatures.md updated

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant