-
Notifications
You must be signed in to change notification settings - Fork 0
Archive Parser
Create channels from Archive.org collections automatically.
./scripts/archive_collection_parser_dialog.shpython3 scripts/archive_collection_parser.py "https://archive.org/details/JHiggens"| File | Location | Size | Purpose |
|---|---|---|---|
| Channel YAML | data/magnum-pi-channel.yaml |
129 KB | 298 episodes |
| Schedule YAML | schedules/magnum-pi-schedule.yml |
54 KB | Playback + breaks |
| Python Parser | scripts/archive_collection_parser.py |
485 lines | CLI tool |
| GUI Script | scripts/archive_collection_parser_dialog.sh |
333 lines | Interactive UI |
- Channel Number: 80
- Total Episodes: 298
- Seasons: 8 + 3 specials
- Breaks: 296 (2-5 minutes each)
- Runtime: ~250-263 hours
- Source: https://archive.org/details/JHiggens
# 1. Import channel
python3 scripts/import_channels.py data/magnum-pi-channel.yaml
# 2. Start server
./start_server.sh
# 3. Tune to channel 80!python3 scripts/archive_collection_parser.py \
"URL" \
--channel-number 80 \
--channel-name "My Channel" \
--min-break 2 \
--max-break 5 \
--output-dir ~/Desktop# Count episodes
grep -c "^ - id: magnum_" data/magnum-pi-channel.yaml
# Result: 298
# Count breaks
grep -c "# .* minute break" schedules/magnum-pi-schedule.yml
# Result: 296
# View sample
head -50 data/magnum-pi-channel.yaml-
Complete Guide:
MAGNUM_PI_CHANNEL_COMPLETE.md -
Tool README:
scripts/ARCHIVE_PARSER_README.md -
Implementation:
ARCHIVE_PARSER_IMPLEMENTATION_SUMMARY.md
# 1-2 minute breaks
python3 scripts/archive_collection_parser.py URL --min-break 1 --max-break 2
# 5-10 minute breaks
python3 scripts/archive_collection_parser.py URL --min-break 5 --max-break 10# Channel 80
python3 scripts/archive_collection_parser.py "URL1" --channel-number 80
# Channel 81
python3 scripts/archive_collection_parser.py "URL2" --channel-number 81- 298 episodes generated
- 296 breaks enforced (2-5 min each)
- Python CLI tool created
- swiftDialog GUI created
- Complete documentation
- Files in correct locations
- Ready to import and stream
Problem: "requests module not found"
Solution: pip3 install requests
Problem: "swiftDialog not found"
Solution: Script auto-installs, or get from https://github.com/swiftDialog/swiftDialog
Problem: "Collection not found"
Solution: Verify URL at archive.org
Your complete Magnum P.I. channel with 298 episodes and 296 enforced breaks (2-5 minutes each) is ready to stream!
Happy Streaming! 📺
StreamTV was encountering errors when trying to stream Magnum P.I. episodes:
httpx.HTTPStatusError: Redirect response '302 Found' for url 'https://archive.org/download/JHiggens/...'
Redirect location: 'https://dn720208.ca.archive.org/0/items/JHiggens/...'
Archive.org returns 302 redirects to direct file servers (e.g., dn720208.ca.archive.org), but the HTTP client wasn't configured to follow these redirects automatically.
Updated streamtv/streaming/stream_manager.py to enable redirect following:
async with httpx.AsyncClient(timeout=config.streaming.timeout) as client:
async with client.stream('GET', stream_url, headers=headers) as response:async with httpx.AsyncClient(timeout=config.streaming.timeout, follow_redirects=True) as client:
async with client.stream('GET', stream_url, headers=headers, follow_redirects=True) as response:-
Line 138: Added
follow_redirects=TruetoAsyncClientinitialization -
Line 129: Added
follow_redirects=Trueto authenticated Archive.org streaming
-
Restart StreamTV server:
# Stop current server (Ctrl+C) # Then restart: ./start_server.sh
-
Test Magnum P.I. channel: Try streaming from Channel 80
After restart, the logs should show:
- ✅ No more "302 Found" errors
- ✅ Successful streaming from Archive.org
- ✅ Content plays without authentication (for public videos)
- Archive.org uses multiple CDN servers
- The 302 redirect points to the optimal server for your location
- This is normal Archive.org behavior
- Following redirects is required for all Archive.org content
Status: ✅ Fixed
Date: December 3, 2025
Affected: All Archive.org streaming (including Magnum P.I. channel)
Successfully created a complete automated system for generating StreamTV channels from Archive.org collections, with strict enforcement of 2-5 minute breaks between episodes. Includes both command-line and beautiful GUI tools.
298 Episodes parsed and configured from https://archive.org/details/JHiggens
| Metric | Value |
|---|---|
| Total Episodes | 298 |
| Seasons | 8 + Specials |
| Channel Number | 80 |
| Inter-Episode Breaks | 296 (strictly enforced) |
| Break Duration | 2-5 minutes (randomized) |
| Total Runtime | ~250-263 hours |
| Generated YAML Lines | 5,433 lines |
- Season 1: 36 episodes
- Season 2: 42 episodes
- Season 3: 44 episodes
- Season 4: 42 episodes
- Season 5: 44 episodes
- Season 6: 40 episodes
- Season 7: 21 episodes
- Season 8: 26 episodes
- Specials: 3 episodes
File: scripts/archive_collection_parser.py (485 lines)
✅ Automatic Metadata Fetching from Archive.org API
✅ Smart Episode Parsing (extracts season/episode from filenames)
✅ Title Cleaning (removes technical jargon)
✅ Complete YAML Generation (channels + schedules)
✅ Strict Break Enforcement (2-5 minutes between episodes)
✅ Multiple Format Support (.mp4, .avi, .mkv, .mov, etc.)
✅ Configurable Settings (channel number, name, break duration)
✅ Error Handling (graceful failures, detailed messages)
python3 scripts/archive_collection_parser.py \
"https://archive.org/details/JHiggens" \
--channel-number 80 \
--channel-name "Magnum P.I. Complete Series" \
--min-break 2 \
--max-break 5 \
--output-dir ~/Desktop- Channel YAML (3,587 lines)
- Schedule YAML (1,846 lines)
- Detailed summary
- Episode statistics
File: scripts/archive_collection_parser_dialog.sh (333 lines)
✅ Beautiful macOS GUI using swiftDialog
✅ User-Friendly Workflow (guided step-by-step)
✅ Auto-Install Dependencies (swiftDialog, Python modules)
✅ Input Validation (URL verification, required fields)
✅ Progress Indicators (real-time processing status)
✅ Results Preview (episode counts, season breakdown)
✅ Automatic File Management (saves to correct directories)
✅ Finder Integration (opens generated files)
-
Welcome Screen
┌─────────────────────────────────────┐ │ Archive.org Collection Parser │ │ │ │ Welcome to StreamTV Parser! │ │ Create channels from Archive.org │ │ │ │ [ Continue ] [ Cancel ] │ └─────────────────────────────────────┘ -
Input Form
Enter Collection Information: ┌─────────────────────────────────────┐ │ Collection URL: [________________] │ │ Channel Number: [80____________] │ │ Channel Name: [________________] │ │ Min Break (min): ▼2 │ │ Max Break (min): ▼5 │ │ │ │ [ Parse Collection ] [ Cancel ] │ └─────────────────────────────────────┘ -
Processing
┌─────────────────────────────────────┐ │ Processing Collection │ │ │ │ Fetching metadata and generating │ │ YAML files... │ │ │ │ [████████████░░░░░░░░░] 65% │ └─────────────────────────────────────┘ -
Results
┌─────────────────────────────────────┐ │ ✅ Collection Parsed Successfully! │ │ │ │ Channel: Magnum P.I. (Channel 80) │ │ Total Episodes: 298 │ │ Break Duration: 2-5 minutes │ │ │ │ Episode Breakdown: │ │ Season 1: 36 episodes │ │ Season 2: 42 episodes │ │ ... │ │ │ │ [ Save Files ] [ Cancel ] │ └─────────────────────────────────────┘ -
Completion
┌─────────────────────────────────────┐ │ 🎉 Channel Created Successfully! │ │ │ │ Files Saved: │ │ • channel-magnum-pi.yaml │ │ • schedule-magnum-pi.yml │ │ │ │ Next Steps: │ │ 1. Review the YAML files │ │ 2. Import: python3 scripts/... │ │ 3. Start StreamTV server │ │ │ │ [ Open in Finder ] [ Done ] │ └─────────────────────────────────────┘
Location: data/magnum-pi-channel.yaml
Size: 129 KB
Lines: 3,587
Contains: All 298 episodes with:
- Unique IDs
- Collection groupings (by season)
- Episode types (episode/special)
- Years and broadcast dates
- Network information
- Runtime estimates
- Source URLs (properly encoded)
- Descriptive notes
Sample Entry:
- id: magnum_s01e01
collection: "Magnum P.I. - Season 1"
type: episode
year: 1981
slot: "S01E01 - Please Don't Eat The Snow In Hawaii (1)"
broadcast_date: 1981-09-08
network: CBS
runtime: PT48M
source: archive
url: https://archive.org/download/JHiggens/...
notes: "Season 1 Episode 1"Location: schedules/magnum-pi-schedule.yml
Size: 54 KB
Lines: 1,846
Contains:
- Content definitions (8 seasons + specials)
- Break collection definition
A complete Magnum P.I. channel has been successfully generated for StreamTV with 298 episodes from the Archive.org collection, including 296 enforced breaks (2-5 minutes) between episodes!
| Metric | Value |
|---|---|
| Total Episodes | 298 |
| Seasons | 8 (plus 3 specials) |
| Channel Number | 80 |
| Inter-Episode Breaks | 296 |
| Break Duration | 2-5 minutes (randomized) |
| Source | Archive.org (JHiggens collection) |
- Specials: 3 episodes (Murder She Wrote crossover + Rockford Files bonus)
- Season 1: 36 episodes
- Season 2: 42 episodes
- Season 3: 44 episodes
- Season 4: 42 episodes
- Season 5: 44 episodes
- Season 6: 40 episodes
- Season 7: 21 episodes
- Season 8: 26 episodes (series finale)
- Episodes: ~238 hours (298 episodes × 48 min average)
- Breaks: ~12-25 hours (296 breaks × 2-5 min)
- Total Content: ~250-263 hours of continuous programming
Location: data/magnum-pi-channel.yaml
Size: 129 KB
Contains: All 298 episodes with complete metadata
Location: schedules/magnum-pi-schedule.yml
Size: 54 KB
Contains: Sequential playback schedule with enforced breaks
Channel 80: Magnum P.I. Complete Series
├── Season 1 (36 episodes)
│ ├── Episode 1: "Please Don't Eat The Snow In Hawaii (1)"
│ ├── [2-5 min break]
│ ├── Episode 2: "Please Don't Eat The Snow In Hawaii (2)"
│ ├── [2-5 min break]
│ └── ... (continues)
├── Season 2 (42 episodes)
├── Season 3 (44 episodes)
├── Season 4 (42 episodes)
├── Season 5 (44 episodes)
├── Season 6 (40 episodes)
├── Season 7 (21 episodes)
├── Season 8 (26 episodes)
└── Specials (3 episodes)
├── Murder She Wrote: Magnum on Ice
└── The Rockford Files bonus episodes
Strictly Enforced: Every episode is followed by a break (except the last)
- Duration: Randomly selected between 2-5 minutes
- Implementation: ISO 8601 duration format (PT2M, PT3M, PT4M, PT5M)
- Total Breaks: 296 breaks throughout the schedule
- Purpose: Mimics traditional TV commercial breaks
Features:
- Fetches metadata from Archive.org API
- Parses 298 video files automatically
- Extracts season/episode information
- Generates complete YAML configurations
- Enforces 2-5 minute breaks between episodes
- Handles multiple video formats (.mp4, .avi, .mkv, etc.)
Usage:
python3 scripts/archive_collection_parser.py \
"https://archive.org/details/JHiggens" \
--channel-number 80 \
--channel-name "Magnum P.I. Complete Series" \
--min-break 2 \
--max-break 5 \
--output-dir /path/to/outputFeatures:
- Beautiful macOS-native GUI using swiftDialog
- Prompts for collection URL
- Customizable channel settings
- Adjustable break durations
- Progress indicators
- Auto-installation of dependencies
- File management and preview
Usage:
./scripts/archive_collection_parser_dialog.shWorkflow:
- Welcome screen with instructions
- Prompt for Archive.org URL
- Configure channel number and name
- Set break duration (min/max)
- Parse collection with progress indicator
- Review results and episode counts
- Save files to StreamTV directories
- Open generated files in Finder
channels:
- number: "80"
name: "Magnum P.I. Complete Series"
group: "Classic Television"
description: "Complete Magnum P.I. series (1980-1988)..."
enabled: true
streams:
- id: magnum_s01e01
collection: "Magnum P.I. - Season 1"
type: episode
year: 1981
slot: "S01E01 - Please Don't Eat The Snow In Hawaii (1)"
broadcast_date: 1981-09-08
network: CBS
runtime: PT48M
source: archive
url: https://archive.org/download/JHiggens/...
notes: "Season 1 Episode 1"name: Magnum P.I. Marathon
description: >-
24/7 marathon with 2-5 minute breaks between episodes.
content:
- key: season1
collection: Magnum P.I. - Season 1
order: chronological
- key: break_short
collection: Inter-Episode Breaks
order: random
sequence:
- key: magnum-marathon
items:
- all: season1
custom_title: "Season 1 Episode 1"
# 3 minute break
- all: break_short
custom_title: "Inter-Episode Break"
duration: PT3M
- all: season1
custom_title: "Season 1 Episode 2"
# 5 minute break
- all: break_short
duration: PT5M
playout:
- sequence: magnum-marathon
- repeat: trueSuccessfully restarted Channel 80 from scratch - cleaned database, regenerated YAML files, and re-imported.
- Deleted Channel 80 from database
- Removed 9 collections (Seasons 1-8 + Specials)
- Removed 1 playlist
- Cleared all associated media items
- Fetched fresh metadata from Archive.org
- Generated new
magnum-pi-channel.yaml(128 KB) - Generated new
magnum-pi-schedule.yml(54 KB) - 298 episodes parsed
- 296 breaks enforced (2-5 minutes)
- Imported fresh data into database
- Created 9 collections
- Configured all 298 episodes
- Channel enabled and ready
- Channel Number: 80
- Name: Magnum P.I. Complete Series
- Group: Classic Television
- Status: ✅ Enabled
- Episodes: 298
- Collections: 9
- Magnum P.I. - Specials: 3 episodes
- Magnum P.I. - Season 1: 36 episodes
- Magnum P.I. - Season 2: 42 episodes
- Magnum P.I. - Season 3: 44 episodes
- Magnum P.I. - Season 4: 42 episodes
- Magnum P.I. - Season 5: 44 episodes
- Magnum P.I. - Season 6: 40 episodes
- Magnum P.I. - Season 7: 21 episodes
- Magnum P.I. - Season 8: 26 episodes
- ✅ Channel 1980: 1980 Lake Placid Winter Olympics
- ✅ Channel 1984: 1984 Sarajevo Winter Olympics
- ✅ Channel 1988: 1988 Calgary Winter Olympics
- ✅ Channel 1992: 1992 Albertville Winter Olympics
- ✅ Channel 1994: 1994 Lillehammer Winter Olympics
- ✅ Channel 80: Magnum P.I. Complete Series ⭐
The channel is in the database, but you need to restart for it to appear in the UI:
# Stop current server (Ctrl+C if running)
# Then restart:
./start_server.shAfter restarting:
- Web UI: http://localhost:8410/channels
- IPTV Playlist: http://localhost:8410/iptv/channels.m3u
- Direct Stream: http://localhost:8410/iptv/stream/80
# Test the stream URL
curl -I http://localhost:8410/iptv/stream/80
# Or open in VLC
vlc http://localhost:8410/iptv/stream/80-
data/magnum-pi-channel.yaml(128 KB) -
schedules/magnum-pi-schedule.yml(54 KB)
- Channel 80 with all 298 episodes
- 9 collections (by season)
- 1 playlist
- All media items linked
- Min Break: 2 minutes
- Max Break: 5 minutes
- Total Breaks: 296 (between all episodes)
- Random Duration: Each break randomly 2-5 minutes
- Type: event (per StreamTV schema)
- Source: archive.org
- Runtime: PT48M (48 minutes average)
- Network: CBS
- Years: 1980-1988
# Count channels
SELECT COUNT(*) FROM channels WHERE number = '80';
# Result: 1
# Count collections
SELECT COUNT(*) FROM collections WHERE name LIKE '%Magnum%';
# Result: 9
# Count total episodes
SELECT COUNT(*) FROM collection_items
WHERE collection_id IN (
SELECT id FROM collections WHERE name LIKE '%Magnum%'
);
# Result: 298ls -lh data/magnum-pi-channel.yaml
# Result: 128K
ls -lh schedules/magnum-pi-schedule.yml
# Result: 54K
grep -c "^ - id: magnum_" data/magnum-pi-channel.yaml
# Result: 298
grep -c "# .* minute break" schedules/magnum-pi-schedule.yml
# Result: 296Your Magnum P.I. channel has been completely restarted from scratch and is ready to stream!
- ✅ Database cleaned
- ✅ Fresh YAML files generated
- ✅ Channel imported successfully
- ✅ All 298 episodes configured
- ✅ 296 breaks enforced
- ✅ Ready for playback
Just restart the server and tune to Channel 80! 📺
Date: December 3, 2025
Status: ✅ Complete
Action: Restart server to see changes
./scripts/archive_collection_parser_dialog.shpython3 scripts/archive_collection_parser.py "https://archive.org/details/JHiggens"| File | Location | Size | Purpose |
|---|---|---|---|
| Channel YAML | data/magnum-pi-channel.yaml |
129 KB | 298 episodes |
| Schedule YAML | schedules/magnum-pi-schedule.yml |
54 KB | Playback + breaks |
| Python Parser | scripts/archive_collection_parser.py |
485 lines | CLI tool |
| GUI Script | scripts/archive_collection_parser_dialog.sh |
333 lines | Interactive UI |
- Channel Number: 80
- Total Episodes: 298
- Seasons: 8 + 3 specials
- Breaks: 296 (2-5 minutes each)
- Runtime: ~250-263 hours
- Source: https://archive.org/details/JHiggens
# 1. Import channel
python3 scripts/import_channels.py data/magnum-pi-channel.yaml
# 2. Start server
./start_server.sh
# 3. Tune to channel 80!python3 scripts/archive_collection_parser.py \
"URL" \
--channel-number 80 \
--channel-name "My Channel" \
--min-break 2 \
--max-break 5 \
--output-dir ~/Desktop# Count episodes
grep -c "^ - id: magnum_" data/magnum-pi-channel.yaml
# Result: 298
# Count breaks
grep -c "# .* minute break" schedules/magnum-pi-schedule.yml
# Result: 296
# View sample
head -50 data/magnum-pi-channel.yaml-
Complete Guide:
MAGNUM_PI_CHANNEL_COMPLETE.md -
Tool README:
scripts/ARCHIVE_PARSER_README.md -
Implementation:
ARCHIVE_PARSER_IMPLEMENTATION_SUMMARY.md
# 1-2 minute breaks
python3 scripts/archive_collection_parser.py URL --min-break 1 --max-break 2
# 5-10 minute breaks
python3 scripts/archive_collection_parser.py URL --min-break 5 --max-break 10# Channel 80
python3 scripts/archive_collection_parser.py "URL1" --channel-number 80
# Channel 81
python3 scripts/archive_collection_parser.py "URL2" --channel-number 81- 298 episodes generated
- 296 breaks enforced (2-5 min each)
- Python CLI tool created
- swiftDialog GUI created
- Complete documentation
- Files in correct locations
- Ready to import and stream
Problem: "requests module not found"
Solution: pip3 install requests
Problem: "swiftDialog not found"
Solution: Script auto-installs, or get from https://github.com/swiftDialog/swiftDialog
Problem: "Collection not found"
Solution: Verify URL at archive.org
Your complete Magnum P.I. channel with 298 episodes and 296 enforced breaks (2-5 minutes each) is ready to stream!
Happy Streaming! 📺
./scripts/archive_collection_parser_dialog.shInputs:
- URL:
https://archive.org/details/JHiggens - Channel Number:
80 - Channel Name:
Magnum P.I. Complete Series - Min Break:
2minutes - Max Break:
5minutes
Result: 298 episodes with 296 enforced breaks
python3 scripts/archive_collection_parser.py \
"https://archive.org/details/TheRockfordFiles" \
--channel-number 81 \
--channel-name "The Rockford Files" \
--min-break 2 \
--max-break 5 \
--output-dir data/Then move and import:
mv data/magnum-pi-schedule.yml schedules/rockford-files-schedule.yml
python3 scripts/import_channels.py data/magnum-pi-channel.yamlpython3 scripts/archive_collection_parser.py \
"https://archive.org/details/feature_films" \
--channel-number 82 \
--channel-name "Classic Cinema" \
--min-break 1 \
--max-break 3 \
--output-dir ~/Desktop/movies_channelMovies are longer, so use shorter breaks (1-3 minutes) between films.
./scripts/archive_collection_parser_dialog.shInputs:
- URL:
https://archive.org/details/documentary_series - Channel Number:
83 - Channel Name:
Documentary Marathon - Min Break:
5minutes - Max Break:
10minutes
Longer breaks give viewers time to process information.
#!/bin/bash
# Create multiple channels
collections=(
"JHiggens:80:Magnum P.I."
"RockfordFiles:81:The Rockford Files"
"MurderSheWrote:82:Murder She Wrote"
)
for item in "${collections[@]}"; do
IFS=':' read -r collection channel_num name <<< "$item"
python3 scripts/archive_collection_parser.py \
"https://archive.org/details/$collection" \
--channel-number "$channel_num" \
--channel-name "$name" \
--min-break 2 \
--max-break 5 \
--output-dir "data/batch_$collection"
echo "✅ Generated channel $channel_num: $name"
done# Test with a small collection first
python3 scripts/archive_collection_parser.py \
"SmallTestCollection" \
--channel-number 99 \
--output-dir /tmp/test_channel
# Review output
ls -lh /tmp/test_channel/
cat /tmp/test_channel/magnum-pi-channel.yaml | head -50# Set min and max both to 0
python3 scripts/archive_collection_parser.py \
"URL" \
--min-break 0 \
--max-break 0# Note: Convert to minutes (0.5 = 30 seconds)
# Currently only supports whole minutes, so use 1 minute minimum
python3 scripts/archive_collection_parser.py \
"URL" \
--min-break 1 \
--max-break 1# Traditional TV commercial breaks (2-3 minutes)
python3 scripts/archive_collection_parser.py \
"URL" \
--min-break 2 \
--max-break 3# Longer breaks for bathroom/snack runs (5-10 minutes)
python3 scripts/archive_collection_parser.py \
"URL" \
--min-break 5 \
--max-break 10# Run with verbose Python
python3 -v scripts/archive_collection_parser.py "URL" 2>&1 | tee debug.log
# Check what was parsed
grep "Found.*video files" debug.log
grep "Total Episodes" debug.log# Count episodes in channel YAML
grep -c "^ - id:" data/magnum-pi-channel.yaml
# Count breaks in schedule YAML
grep -c "# .* minute break" schedules/magnum-pi-schedule.yml# See how breaks are distributed
grep "duration: PT" schedules/magnum-pi-schedule.yml | \
sed 's/.*PT\([0-9]*\)M/\1/' | \
sort -n | \
uniq -c
# Output example:
# 58 2
# 62 3
---
## Related Pages
- [Scripts and Tools](Scripts-and-Tools)
- [Home](Home)