POC changes for making search plugin hot reloadable

[pgtgrly]

Summary

This is a Proof of Concept demonstrating dynamic loading of search plugins with custom query types at runtime without cluster restart. Currently supports queries only and requires security hardening before production use.

Related PR: #19809

What This POC Includes

• Dynamic loading of SearchPlugin implementations via REST API
• Runtime registration of custom query types
• Thread-safe registry propagation through query parsing chain
• Immediate query availability after plugin load
• Test plugin demonstrating the capability

What This POC Does NOT Include

Critical Missing Features:

• Security Manager integration (currently disabled)
• Proper file permission restrictions (wildcard permissions used)
• Cluster-wide coordination (single node only)
• Support for aggregations, suggesters, or other SearchPlugin components
• Plugin removal/unload capability
• Query cache invalidation
• Rollback mechanism for failed updates
• Plugin state persistence across restarts
• Production-grade error handling and validation

Problem

OpenSearch currently requires full cluster restarts to add or update search plugins, causing operational overhead and deployment complexity.

Solution

Dynamic plugin lifecycle with immediate query availability:

• Load plugin JARs via REST API
• Extract and register query specifications
• Propagate updated registries through parsing chain
• Use custom queries immediately

Key Changes

New Components:

• SearchPluginHotReloadService - Orchestrates plugin loading and registry updates
• LoadSearchPluginsAction with REST endpoint: POST /_search_plugins/load

Modified Components:

• SearchModule - Added dynamic query tracking (ConcurrentHashMap)
• IndicesService, IndexService, AbstractHttpServerTransport - Changed registry references from final to volatile, added update methods
• PluginsService - Added loadPluginDynamically() method
• ActionModule, Node - Wired new service into dependency graph
• Security.java, security.policy - Temporarily disabled/relaxed for POC

Architecture

Registry propagation chain:

SearchModule → NamedXContentRegistry/NamedWriteableRegistry → IndicesService 
→ IndexService → AbstractHttpServerTransport → Query parsing

Thread safety via volatile references and atomic registry swaps.

API Usage

# Load plugin
POST /_search_plugins/load?plugin_path=/path/to/plugin

# Use custom query immediately
POST /my-index/_search
{
  "query": {
    "custom_query_type": { ... }
  }
}

Testing

• End-to-end integration tests with TestSearchPlugin
• Validated query execution with hot-reloaded components
• Confirmed thread safety and no memory leaks in POC testing

Breaking Changes

None. Additive and backward compatible.

[github-actions]

❌ Gradle check result for 9349782: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?