[AINode] Refactoring of Model Storage, Loading, and Inference Pipeline

[CRZbulabula]

This PR introduces significant improvements in the model storage, loading, and inference pipeline management for better extensibility, efficiency, and ease of use. The changes include the refactoring of model storage to support a wider range of models, streamlining the model loading process, and the introduction of a unified inference pipeline. These improvements aim to optimize model management, reduce memory usage, and enhance the overall inference workflow.

• Model Storage Refactoring

  • Extended Support for Models: The system now supports not only built-in models like TimerXL and Sundial but also allows the integration of fine-tuned and user-defined models.
  • Unified Model Management: A new model management system enables model registration, deletion, and loading from both local paths and Hugging Face.
  • Code Optimization: Redundant code from previous versions has been removed, and hard-coded model management has been replaced by a more flexible approach that integrates seamlessly with the Hugging Face Transformers ecosystem.

• Model Loading Refactoring

  • Simplified Model Loading: The previous custom loading logic with complex if...else... conditions has been replaced by a unified model loading interface, simplifying the process.
  • Automatic Model Type Detection: The system now automatically detects the model type and selects the appropriate loading method, supporting models from Transformers, sktime, and PyTorch.
  • Lazy Loading: The PR introduces lazy loading for Python modules, eliminating the need to load multiple modules at startup, reducing initialization time and memory consumption.

• Inference Pipeline Addition

  • Unified Inference Workflow: The introduction of the Inference Pipeline encapsulates the entire model inference process, offering a standardized interface for preprocessing, inference, and post-processing.
  • Support for Multiple Tasks: The pipeline is versatile, supporting various inference tasks such as prediction, classification, and dialogue-based tasks.

[codecov]

Codecov Report

❌ Patch coverage is 12.50000% with 196 lines in your changes missing coverage. Please review.
✅ Project coverage is 38.98%. Comparing base (1746cdb) to head (5cc12be).
⚠️ Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
...ache/iotdb/db/protocol/client/an/AINodeClient.java	1.88%	104 Missing ⚠️
...g/apache/iotdb/commons/model/ModelInformation.java	0.00%	21 Missing ⚠️
...ion/config/executor/ClusterConfigTaskExecutor.java	5.88%	16 Missing ⚠️
...apache/iotdb/commons/client/ClientPoolFactory.java	0.00%	14 Missing ⚠️
...relational/function/tvf/ForecastTableFunction.java	11.11%	8 Missing ⚠️
...he/iotdb/db/queryengine/plan/udf/UDTFForecast.java	0.00%	7 Missing ⚠️
...db/db/queryengine/plan/analyze/AnalyzeVisitor.java	0.00%	5 Missing ⚠️
...plan/parameter/model/ModelInferenceDescriptor.java	0.00%	5 Missing ⚠️
...ode/procedure/impl/node/RemoveAINodeProcedure.java	0.00%	3 Missing ⚠️
...ecution/operator/process/ai/InferenceOperator.java	0.00%	3 Missing ⚠️
... and 6 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #16819      +/-   ##
============================================
+ Coverage     38.87%   38.98%   +0.11%     
  Complexity      207      207              
============================================
  Files          5022     5009      -13     
  Lines        333113   332063    -1050     
  Branches      42390    42260     -130     
============================================
- Hits         129488   129449      -39     
+ Misses       203625   202614    -1011     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[CRZbulabula]

PTAL.

[sonarqubecloud]

Quality Gate passed

Issues
27 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
3.8% Duplication on New Code

See analysis details on SonarQube Cloud