Pr/04 frontend metadata #69

codihuston · 2026-01-29T03:35:52Z

Summary

Add React frontend with video player, gallery, and artifact viewers. Implement metadata extraction pipeline for video file timestamps and duration.

Changes

Testing

Tests pass locally
New tests added (if applicable)
Manual testing completed

Related Issues

N/A

Screenshots (if applicable)

N/A

6b877df fix(frontend): back button in video player goes back to main menu
559bc2e feat(ui): add metadata viewer and task requeue functionality
af79662 feat(metadata): implement video metadata extraction with reverse geocoding
28c0d3e feat(projection-sync): add metadata artifact GPS projection sync
c4e4545 feat(artifact-transformer): add metadata_extraction support
0118f70 feat(metadata-extraction): add metadata_extraction to active task types
1eeb3cf feat(metadata): register metadata_extraction task type in task registry
1b293ba feat(metadata-extraction): implement metadata extraction pipeline
16e91e5 feat(metadata): create MetadataV1 schema and register with SchemaRegistry
764ac14 fix(places): places are now labeled properly
58e5c74 feat(frontend): improve video player
4e20065 feat(backend): add multi-lingual tasks
63d07f6 feat(frontend): add poc for gallery and single player
de45b80 feat(ml-service): significantly improve inference perf
a4a85a1 fix(backend): reconciler enqueues unique tasks
5137271 fix(backend): jump player backwards navigation
726c425 feat(backend): add task status endpoint
fec94f3 refactor(backend): update config shape and thersholds
f1bc849 fix(face-detection): improve detection interval
f697809 fix(backend): case insensitiivty for video discovery exts
20aae9e fix(frontend): face detection poc uses new artifacts endpoint
8ef8bf2 fix(ml-service): fix image deps
3f4409f feat(frontend): add poc for scene detection jumping
fa44915 feat(backend): add video stream endpoint
1c8e38d fix(scene-detection): set threshold to 0.7

Detailed Changes

Frontend

Add VideoPlayer component with timeline scrubbing and playback controls
Add VideoGallery component for browsing videos
Add artifact viewer components:
- OCRViewer / OCROverlay - text detection display
- TranscriptViewer - speech transcription with timestamps
- ObjectDetectionViewer / ObjectDetectionOverlay - YOLO detections
- SceneDetectionViewer - scene boundary display
- PlaceDetectionViewer - location detection display
Add TaskStatusViewer for real-time task progress monitoring
Add MetadataViewer for video metadata display
Implement scene detection jumping POC
Implement face detection POC with artifacts endpoint

ML Service

Implement metadata extraction service using ffprobe and EXIF data
Extract file_created_at timestamp from video files
Extract duration from video container
Add metadata_extraction task type to task registry
Significantly improve inference performance
Add reverse geocoding for GPS coordinates

Backend

Add file_created_at field to video domain model
Update video repository to persist metadata
Add video stream endpoint
Add task status endpoint
Fix reconciler to enqueue unique tasks
Add multi-lingual task support

- Uses ffmpeg and supports range requests

- pytorch is working again

- Supported via config - Samples every 1 second

- Object detection no longer run per frame, instead per interval like face detection

- No global search yet - Jumps supported in single video view (transcript, scenes, objects, ocr, places, faces) - Shows task status - Renders boxes on faces and objects

- Tasks now have a language property that is propagated down into the artifact as well - This enables us to run multiple language models and tasks per language against a video (ocr, transcription) - Add task status endpoint for a given video - Fix backend dockerfile, it was built like the ml-service for some reason - Fix typo in faces file: yolov8n-face.pt

- Show video task status - Show transcript, ocr, with selection profile options (latest per language), specific runs - Add ocr overlay - Add faces tab - Add re-render when confidence changes in jump navigator

- The accuracy still seems a bit off, but, we have usable data now anyway

…stry - Create MetadataV1 Pydantic schema with all metadata fields (GPS, camera, file, temporal, image) - Register 'video.metadata' artifact type with SchemaRegistry - Add comprehensive schema validation tests (19 tests covering all fields and edge cases) - Update schema initialization to register MetadataV1 during startup - Add new task 2.5 for capturing file creation timestamp with fallback chain All tests passing. Ready for metadata extraction implementation.

- Add video_locations projection table for GPS normalization - Add file_created_at column to videos table for temporal queries - Implement MetadataExtractor using pyexiftool for standardized field extraction - Add metadata_extraction task support to ML service task handler - Extract Composite fields with proper type conversion - Set producer=pyexiftool, model_profile=balanced for metadata artifacts Validates: Requirements 1.1-1.5, 2.1-2.6, 3.1-3.5, 4.1-4.4, 6.1-6.4

- Add metadata_extraction to TASK_REGISTRY with LanguageMode.NONE - Mark as language-agnostic task type (no language parameter required) - Add metadata_extraction worker configuration to content_creator.json - 2 concurrent workers, priority 1, CPU resource - Update task status for task 4 to completed Validates Requirements 4.1, 4.3

- Add 'metadata_extraction' to ACTIVE_TASK_TYPES in VideoDiscoveryService - Add empty default configuration for metadata_extraction task - Metadata extraction tasks will now be created and enqueued on video discovery - Tasks are created with language=NULL (language-agnostic) - Producer and model_profile metadata come from ML worker result, not task config Validates Requirements 4.1, 4.2, 4.3, 4.4

- Add MetadataV1 schema to ARTIFACT_SCHEMA_MAP - Map metadata_extraction task type to video.metadata artifact type - Handle metadata as single dict (wrap as list for uniform processing) - Extract timing: span_start_ms=0, span_end_ms=duration_ms - Add comprehensive unit tests for metadata transformation Validates Requirements 2.1, 2.4, 2.5

- Add video.metadata artifact type to sync_artifact dispatcher - Implement _sync_video_metadata method for GPS projection - Extract and validate GPS coordinates from metadata payload - Create video_locations entries for geo-spatial queries - Handle missing GPS gracefully (skip projection) - Support both PostgreSQL and SQLite - Add comprehensive test coverage (6 new tests) Validates Requirements 2.1, 2.2, 2.5, 2.6

…oding - Add file_created_at to Video model with EXIF/filesystem fallback - Create reverse geocoding service for GPS coordinate normalization - Update video_locations projection to include country/state/city - Add /api/v1/videos/{videoId}/location endpoint - Fix metadata extraction to use ExifToolHelper API - Update ml-service to populate file_created_at and create metadata artifacts - Update video_locations migration to use video_id unique constraint - Add geopy dependency to backend and ml-service - Update Dockerfile.ml-service to install pre-built exiftool binary

- Add MetadataViewer component with location display (coordinates, altitude, reverse-geocoded location) - Add metadata tab to VideoPlayer with video name display in header - Add requeue menu to failed tasks in TaskStatusViewer - Remove video.metadata from jump navigation dropdown - Fix projection sync test to use video_id column - Update task spec documentation

codihuston added 26 commits January 28, 2026 21:28

fix(scene-detection): set threshold to 0.7

143f276

feat(backend): add video stream endpoint

4eca4c2

- Uses ffmpeg and supports range requests

feat(frontend): add poc for scene detection jumping

1d6d29d

fix(ml-service): fix image deps

4d6972b

- pytorch is working again

fix(frontend): face detection poc uses new artifacts endpoint

63e644d

fix(backend): case insensitiivty for video discovery exts

d8d5047

fix(face-detection): improve detection interval

68dd4c7

- Supported via config - Samples every 1 second

refactor(backend): update config shape and thersholds

fede327

- Object detection no longer run per frame, instead per interval like face detection

feat(backend): add task status endpoint

30fa2fb

fix(backend): jump player backwards navigation

8fb54a3

fix(backend): reconciler enqueues unique tasks

e355d65

feat(ml-service): significantly improve inference perf

4792f8b

feat(frontend): add poc for gallery and single player

67aada7

- No global search yet - Jumps supported in single video view (transcript, scenes, objects, ocr, places, faces) - Shows task status - Renders boxes on faces and objects

feat(frontend): improve video player

ad88732

- Show video task status - Show transcript, ocr, with selection profile options (latest per language), specific runs - Add ocr overlay - Add faces tab - Add re-render when confidence changes in jump navigator

fix(places): places are now labeled properly

2acc67e

- The accuracy still seems a bit off, but, we have usable data now anyway

fix(frontend): back button in video player goes back to main menu

04cbca0

fix(dev): fix missing volume mount

340921e

codihuston force-pushed the pr/04-frontend-metadata branch from 405130c to 340921e Compare January 29, 2026 03:42

codihuston merged commit ebfd7f9 into main Jan 29, 2026
0 of 2 checks passed

codihuston deleted the pr/04-frontend-metadata branch January 31, 2026 06:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pr/04 frontend metadata #69

Pr/04 frontend metadata #69

codihuston commented Jan 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Pr/04 frontend metadata #69

Pr/04 frontend metadata #69

Conversation

codihuston commented Jan 29, 2026

Summary

Changes

Testing

Related Issues

Screenshots (if applicable)

Detailed Changes

Frontend

ML Service

Backend

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant