MQTT Daemon Major upgrade including SSL support and many enhancements

[Petrus79]

Merry Christmas Pull Request to N3rdix/pyHPSUmqtt Master

🎯 Pull Request Title

Comprehensive SocketCAN Improvements and MQTT Enhancements

📋 Summary

This pull request contributes significant improvements to the pyHPSUmqtt project, focusing on SocketCAN reliability, MQTT functionality enhancements, and Python compatibility fixes. These changes have been thoroughly tested and address several critical issues while maintaining backward compatibility.

🔧 Key Improvements

1. SocketCAN Communication Enhancements

• Fixed CanPI loop delay issues: Properly configurable timeout from config file
• Improved CANPI section handling: Fixed issue where CANPI configuration wasn't being used from pyhpsu.conf
• Enhanced retry mechanism: Added configurable delay for CanPI retry operations
• Performance optimization: Removed unnecessary sleep delays while maintaining configurability

2. MQTT System Improvements

• MQTT Daemon V2: Complete rewrite of MQTT daemon functionality
• Session Management: Fixed MQTT session conflicts in daemon mode
• Automatic Mode Selection: MQTTDAEMON output mode automatically enabled when --mqtt_daemon parameter is used
• Enhanced Plugin Architecture: Improved MQTT plugin with better error handling

3. Python Compatibility & Code Quality

• Python 3.12+ Support: Fixed compatibility issues with newer Python versions
• Python 3.9+ Support: Ensured backward compatibility with Python 3.9+
• Code Review Fixes: Addressed various code quality findings from automated reviews
• Duplicate Logging Fix: Resolved duplicate logging issues
• Loglevel Changes: Moved many logs from Loglevel INFO to DEBUG to get clean logs in Loglevel mode INFO.

4. SSL/TLS Security Enhancement

• Complete SSL Support: Full SSL/TLS encryption for MQTT connections
• Certificate Management: Support for CA certificates, client certificates, and private keys
• Security Configuration: Flexible SSL options in configuration file
• Enterprise Ready: Production-grade security implementation

5. Heat Pump Mode Mapping Improvements

• Fixed Unknown Mode Values: Added proper descriptive names for heat pump operation states
• Enhanced Mode Visibility: Replaced generic numeric codes with meaningful mode names
• MQTT Integration: Heat pump modes now display as "defrost" and "water heating" instead of "3" and "4"
• Home Assistant Compatibility: Improved state reporting for home automation systems

📊 Changes Overview

Files Modified:

• HPSU/canpi.py - SocketCAN driver improvements
• HPSU/HPSU.py - Core HPSU logic enhancements
• HPSU/mqtt_daemon.py - Complete MQTT daemon rewrite
• HPSU/plugins/mqtt.py - MQTT plugin improvements
• pyHPSU.py - Main script optimizations and SSL integration
• etc/pyHPSU/pyhpsu.conf - SSL configuration options added
• etc/pyHPSU/commands_hpsu.json - Heat pump mode mapping improvements
• .gitignore - SSL certificate exclusions

New Files Added:

• SSL_README.md - Comprehensive SSL/TLS configuration guide

Statistics:

• Commits: 10+ significant commits
• Lines Added: ~800+ lines
• Lines Modified: ~300+ lines
• Features Added: SSL/TLS, MQTT daemon v2, improved SocketCAN handling, heat pump mode fixes

🚀 Technical Details

SocketCAN Reliability Improvements

# Before: Fixed delays causing performance issues
time.sleep(1.0)  # Hard-coded delay

# After: Configurable timeout from config
timeout = self.timeout  # From pyhpsu.conf [CANPI] section

MQTT Daemon V2 Features

• Session Isolation: Prevents conflicts between daemon and plugin modes
• Automatic Configuration: Streamlined setup process
• Enhanced Error Handling: Better debugging and reliability
• Memory Optimization: Improved object lifecycle management

SSL/TLS Implementation

[MQTT]
SSL_ENABLED = True
SSL_CA_CERT = /path/to/ca.crt
SSL_CERTFILE = /path/to/client.crt
SSL_KEYFILE = /path/to/client.key
SSL_INSECURE = False

Heat Pump Mode Improvements

// Before: Generic numeric codes
"value_code" : {
    "standby" :"0",
    "heating" : "1", 
    "cooling" : "2",
    "3": "3"        // ← Unclear what mode 3 represents
}

// After: Descriptive mode names
"value_code" : {
    "standby" :"0",
    "heating" : "1",
    "cooling" : "2", 
    "defrost": "3",         // ← Clear defrost mode
    "water heating" : "4"   // ← Clear water heating mode
}

✅ Testing & Validation

Tested Configurations:

• ✅ SocketCAN (PYCAN): Primary testing platform
• ✅ ELM327 Serial: Backward compatibility verified
• ✅ MQTT Daemon Mode: Full functionality tested
• ✅ SSL/TLS Connections: Secure MQTT communication verified
• ✅ Python 3.9-3.12: Cross-version compatibility confirmed

Test Commands:

# MQTT Daemon with SSL
python3 pyHPSU.py --mqtt_daemon --log_level DEBUG -a -o MQTT

# SocketCAN mode setting (previously broken, now working)
mosquitto_pub -t "rotex/command/mode_01" -m "heat"
mosquitto_pub -t "rotex/command/mode_01" -m "3"

🛡️ Backward Compatibility

Maintained Compatibility:

• ✅ All existing configuration files work unchanged
• ✅ All CAN drivers (ELM327, TCP, EMU) function as before
• ✅ Existing MQTT topics and commands preserved
• ✅ Non-SSL MQTT connections continue to work
• ✅ All systemd service configurations remain valid

Migration Path:

• No breaking changes - all improvements are additive
• Optional SSL - can be enabled when needed
• Configurable delays - existing behavior preserved by default

🎯 Benefits to N3rdix/pyHPSUmqtt

User Experience:

• Improved Reliability: SocketCAN communication more stable
• Enhanced Security: Enterprise-grade SSL/TLS encryption
• Better Performance: Optimized timing and reduced delays
• Easier Setup: Automatic daemon mode configuration
• Clearer Mode Display: Heat pump modes show descriptive names instead of numbers

Developer Experience:

• Code Quality: Fixed various code review findings
• Documentation: Comprehensive SSL setup guide
• Maintainability: Cleaner MQTT daemon architecture
• Future-Proof: Python 3.12+ compatibility

Enterprise Adoption:

• Security Compliance: SSL/TLS support for enterprise environments
• Scalability: Improved MQTT daemon performance
• Configuration Management: Better config file handling
• Production Ready: Thoroughly tested improvements

📋 Merge Considerations

Low Risk Changes:

• All improvements are additive and non-breaking
• Extensive testing on SocketCAN (primary use case)
• Backward compatibility maintained for all interfaces
• Optional features can be enabled as needed

High Value Additions:

• SSL/TLS security brings enterprise capabilities
• SocketCAN improvements address real user pain points
• MQTT daemon v2 provides better reliability
• Python compatibility ensures future usability

🔍 Code Review Notes

Key Areas for Review:

1. SSL Implementation: Review security implementation in mqtt.py and pyHPSU.py
2. SocketCAN Changes: Verify canpi.py improvements don't break existing setups
3. MQTT Daemon: Check mqtt_daemon.py architecture and session handling
4. Configuration: Ensure pyhpsu.conf changes are backward compatible

Testing Recommendations:

1. Test with existing SocketCAN setups
2. Verify ELM327 serial interface still works
3. Test MQTT daemon with and without SSL
4. Confirm configuration file compatibility

📚 Documentation

Included Documentation:

• SSL_README.md: Complete SSL/TLS setup guide with examples
• Configuration Examples: SSL settings in pyhpsu.conf
• Security Best Practices: Certificate management guidance

Future Documentation Needs:

• Update main README.md with SSL capabilities
• Add MQTT daemon v2 usage examples
• Document SocketCAN performance improvements

---

🤝 Contribution Statement

This pull request represents a comprehensive improvement to pyHPSUmqtt with focus on:

• Reliability: Better SocketCAN communication
• Security: Enterprise-grade SSL/TLS support
• Performance: Optimized timing and reduced delays
• Compatibility: Python 3.9-3.12 support
• Quality: Code review fixes and improved architecture

All changes have been thoroughly tested with SocketCAN setups and maintain full backward compatibility. The improvements address real user pain points while adding valuable enterprise features.

Ready for merge after review and testing confirmation.