Implement Node Metadata API

[shenning00]

Overview

Implement a unified Node Metadata API to address two critical issues:

1. Serialization Problem: Port metadata and defaults are lost during graph save/restore
2. UI Introspection Problem: Tools cannot query port configuration from running nodes

Description

This is a comprehensive feature that introduces metadata querying capabilities, port constraint handling, and improved graph serialization to preserve port definitions. The implementation is focused on flow-core C++ library only, without language binding considerations.

Related Issues

• Port metadata introspection requirements
• Graph serialization losing port configuration
• Port constraint definitions and validation

Key Changes

• Add PortMetadata and PortDefinitions types
• Extend Port class to track defaults and constraints
• Add metadata query methods to Node class
• Implement constraint handling (set, validate, serialize)
• Update Graph and Module serialization

Benefits

✅ Graphs save and restore with complete port configuration
✅ UI tools can query port metadata from running node instances
✅ Ports can define constraints (numeric ranges, enums, patterns, custom)
✅ Constraints are serialized and available at load time
✅ Optional constraint validation at runtime
✅ Backward compatible with existing graphs
✅ Consistent metadata API for serialization and introspection

Constraint Handling

What Are Constraints?

Constraints are optional validation rules for port data. They specify acceptable values/ranges and are stored in the port metadata. Constraints enable:

• UI Validation: Visual editors can generate appropriate input controls (sliders, dropdowns, text fields with validation)
• Server-side Validation: Nodes can validate input data before computation
• Schema Documentation: Ports declare their valid data ranges/types

Supported Constraint Types

Numeric Constraints (int, float, double):

• min, max, step for range and granularity

String Constraints (std::string):

• min_length, max_length for length validation
• pattern for regex validation

Enumeration Constraints (any type):

• allowed_values for discrete set of valid values

Collection Constraints (arrays, vectors):

• min_items, max_items for collection size

Custom Constraints:

• Extensible via custom_rule_name fields for domain-specific rules

Setting Constraints

Method 1: At Port Creation (Recommended):

AddInputWithConstraints("vbr", "Variable Bitrate", "int",
                       MakeNodeData<int>(128),
                       json{{"min", 0}, {"max", 320}, {"step", 1}});

Method 2: Via Port API:

GetInputPort("quality")->SetConstraints(
    json{{"allowed_values", {"low", "medium", "high"}}});

When Constraints Are Applied

• Serialization: Constraints are automatically included in port_definitions
• Restoration: Constraints are loaded when graphs are restored
• UI Integration: Constraints available via GetInputPortMetadata()
• Runtime Validation: Nodes can check constraints in Compute() (optional)

Design Principles

• Optional: Not all ports need constraints
• Immutable: Set once at creation, cannot be modified
• Serialized: Automatically persisted in port definitions
• Non-enforcing: Framework provides constraints, validation is optional

Development Plan

The implementation is structured in 4 phases over 6-7 weeks:

Phase 1: Foundation & Port Metadata (Weeks 1-2)

• Implement PortMetadata and PortDefinitions types
• Enhance Port class with metadata tracking
• Add unit tests
• Effort: 1 week

Phase 2: Node Metadata API & Constraints (Weeks 3-4)

• Add metadata query methods to Node
• Update Node::Save()/Restore() to include port_definitions
• Implement constraint infrastructure (NEW)
• Add serialization hooks for custom defaults
• Update FunctionNode for constraint support
• Effort: 2 weeks

Phase 3: Graph & Module Updates (Weeks 5-6)

• Update Graph and Module serialization
• Implement port definition validation
• Add constraint validation logic (NEW)
• Effort: 2 weeks

Phase 4: Documentation & Polish (Week 7)

• Complete API documentation
• Constraint usage guide (NEW)
• Migration guides for custom nodes
• Example applications
• Effort: 1 week

Deliverables

New APIs

C++ Public Methods:

• json GetInputPortMetadata(const IndexableName& key) const
• json GetAllInputPortsMetadata() const
• json GetOutputPortMetadata(const IndexableName& key) const
• json GetAllOutputPortsMetadata() const
• json GetPortDefinitions() const
• json GetNodeMetadata() const

C++ Protected Methods:

• void AddInputWithConstraints(..., const json& constraints)
• void AddOutputWithConstraints(..., const json& constraints)
• virtual json SaveInputDefaults() const
• virtual void RestoreInputDefaults(const json&)

Port Methods:

• bool HasDefaultValue() const
• const SharedNodeData& GetDefaultValue() const
• std::optional<const json&> GetConstraints() const
• void SetConstraints(json constraints)

New Types

• PortMetadata struct with constraint support
• PortDefinitions struct

Enhanced JSON Format

Saved graphs now include:

{
  "port_definitions": {
    "inputs": [
      {
        "key": "bitrate",
        "caption": "Bitrate (kbps)",
        "type": "int",
        "required": false,
        "has_default": true,
        "default_value": "128",
        "constraints": {
          "min": 0,
          "max": 320,
          "step": 1,
          "description": "Audio bitrate in kilobits per second"
        },
        "metadata": null
      }
    ]
  }
}

Impacted Components

• Port class (constraint tracking and management)
• Node class (new query methods, constraint API, updated serialization)
• Graph class (serialization and validation updates)
• Module class (serialization updates)
• FunctionNode class (constraint support)

Success Criteria

• ✅ All new C++ methods fully implemented and tested
• ✅ Constraint infrastructure working (set, store, serialize, restore)
• ✅ Backward compatible with existing graphs
• ✅ Complete documentation and migration guides
• ✅ All tests passing (>90% coverage)
• ✅ Example code demonstrating constraint usage
• ✅ Graphs save and restore with port constraints
• ✅ Constraint validation pattern documented

Acceptance

This issue will be closed when all phases are complete, comprehensive tests pass, and documentation is finalized.

---

Estimated Effort: 6-7 weeks
Team: Lead developer, 1-2 core developers, QA engineer, documentation writer
Dependencies: None (leverages existing infrastructure)
Risk Level: Medium
Scope: flow-core C++ library (no FFI or language bindings)

For full details, see metadata_api.md in the repository.