Modernization Roadmap for main branch #17
Description
Overview
Full modernization of codebase to modern Python/TensorFlow stack.
Target Stack
- Python: 3.11+
- TensorFlow: 2.17+
- Pandas: 2.2+ (breaking changes from 2.0)
- NumPy: 2.0+ (breaking changes)
- Scikit-learn: 1.5+
- Federated Learning: Flower (flwr) instead of TFF
Breaking Changes to Address
1. Pandas 2.0+ Migration
DataFrame.append()removed → Usepd.concat()- Impact: ~10 locations
- Files: train_og.py, test.py, classification scripts
2. Keras 3.0 Migration
- Standalone
kerasdeprecated - Options:
tensorflow.kerasor newkeras3.0 package - Impact: All imports, model loading/saving
.h5format may need migration to.keras
3. TensorFlow 2.17+ Changes
- Optimizer API changes
- Model saving format changes
- Callback signatures may differ
4. NumPy 2.0 Changes
- Type system changes
- Some deprecated functions removed
- May impact array operations
5. Federated Learning Rewrite
- Replace TensorFlow Federated with Flower framework
- TFF 0.40.0 has impossible dependencies (jaxlib conflicts)
- Flower is better documented, actively maintained
- Original FL code never used actual botnet data anyway
Migration Strategy
- Fix deprecations first (pandas, keras imports)
- Update TensorFlow code for 2.17+ API
- Test classification and anomaly detection modules
- Rewrite FL implementation with Flower (new code, not port)
- Add proper unit tests
- Add CI/CD pipeline
Dependencies
- Blocked by: CRITICAL: Data leakage in scaler fitting #13 (data leakage - should fix first)
- Related: Fix deprecated pandas.append() calls #14, Mixed keras imports causing version conflicts #15, Test/train data split overlap concern #16
Branch
Work will be done on main branch after archive-2020-fixed is complete.
References
- Flower docs: https://flower.ai/docs/
- TensorFlow 2.17 migration: https://www.tensorflow.org/guide/migrate
- Pandas 2.0 migration: https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html