Add QSO logging, better highlight logic, migrate to QSO parser in Wsj…

…txUtils.Messages relates #1

README.md

@@ -4,7 +4,7 @@ A .NET 6 console application that highlights callsigns within [WSJT-X](https://p
 
 WjtxUtils.Searchlight will download reception reports from PSK Reporter for the callsigns of any connected WSJT-X client. These reception reports are then correlated with recent decodes and highlighted within WSJT-X. The background color of the highlighted callsign is mapped to the range of received reception report SNR values and used to indicate the relative strength of the received signal.
 
-When a logged QSO occurs, that callsign will be highlighted for the duration of the application. There is no persistence of logged QSOs, and all highlights are cleared on application exit.
+When a logged QSO occurs, that callsign will be highlighted for the duration of the application and can be optionally logged to a file to allow for tracking logged QSOs between sessions.
 
 There is a countdown period when the application starts until requesting the first reception report, which defaults to 5 minutes. This period is an excellent time to call CQ for a while to seed the initial reception report.
 
@@ -52,13 +52,14 @@ To use 3rd party software ([GridTracker](https://gridtracker.org/grid-tracker/))
     "Port": 2237
   }
 ```
-The color options used for reception reports and logged QSOs are altered through the Palette section. ```ReceptionReportBackgroundColors``` is a list of colors, [a gradient](https://colordesigner.io/gradient-generator), used as the background color for reception reports. The first color in the list represents the weakest SNR report values and the last the strongest SNR values. ```ReceptionReportForegroundColor``` controls the color used for highlighted text (callsigns). Both ```ContactedBackgroundColor``` and ```ContactedForegroundColor``` are used for logged QSOs.
+The color options used for reception reports and logged QSOs are altered through the Palette section. ```ReceptionReportBackgroundColors``` is a list of colors, [a gradient](https://colordesigner.io/gradient-generator), used as the background color for reception reports. The first color in the list represents the weakest SNR report values and the last the strongest SNR values. ```ReceptionReportForegroundColor``` controls the color used for highlighted text (callsigns). Both ```ContactedBackgroundColor``` and ```ContactedForegroundColor``` are used for logged QSOs. ```HighlightCallsignsPeriodSeconds``` is the period at which callsigns will be correlated and highlighted based on the current reception report.
 ```json
 "Palette": {
     "ReceptionReportBackgroundColors": [ "#114397", "#453f99", "#653897", "#812e91", "#991f87", "#ae027a", "#be006a", "#cb0058", "#d30044", "#d7002e" ],
     "ReceptionReportForegroundColor": "#ffff00",
     "ContactedBackgroundColor": "#000000",
-    "ContactedForegroundColor": "#ffff00"
+    "ContactedForegroundColor": "#ffff00",
+    "HighlightCallsignsPeriodSeconds": 5
   }
 ```
 Reception report options are altered through the ```PskReporter``` section. ```ReportWindowSeconds``` is a negative number in seconds to indicate how much data to retrieve. This value cannot be more than 24 hours and defaults to -900 seconds or the previous 15 minutes, which should be enough data for current band conditions. ```ReportRetrievalPeriodSeconds``` controls how often reception reports are retrieved. Philip from [PSK Reporter](https://pskreporter.info/) has asked to limit requests to once every five minutes. IMHO Philip does a considerable service to the ham radio community with this data, don't abuse it.
@@ -68,6 +69,13 @@ Reception report options are altered through the ```PskReporter``` section. ```R
     "ReportRetrievalPeriodSeconds": 300
   }
 ```
+The ```LoggedQsos``` section allows adding an optional log file to maintain logged QSOs across searchlight sessions. Set a ```LogFilePath```  to enable QSO logging. ```QsoManagerBehavior``` controls how logged QSOs are highlighted, once per band or once per band and mode.
+```json
+"LoggedQsos": {
+    "LogFilePath": "qsolog.txt",
+    "QsoManagerBehavior": "OncePerBand"
+  }
+```
 Console and file logging output is controlled through the ```Serilog``` section. Please see the [Serilog documentation](https://github.com/serilog/serilog-settings-configuration) for details.
 ```json
 "Serilog": {
@@ -79,14 +87,6 @@ Console and file logging output is controlled through the ```Serilog``` section.
           "theme": "Serilog.Sinks.SystemConsole.Themes.AnsiConsoleTheme::Code, Serilog.Sinks.Console",
           "outputTemplate": "[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj} {NewLine}{Exception}"
         }
-      },
-      {
-        "Name": "File",
-        "Args": {
-          "path": "searchlight-log.txt",
-          "rollingInterval": "Day",
-          "outputTemplate": "[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj} {NewLine}{Exception}"
-        }
       }
     ]
   }

src/WsjtxUtils.Searchlight.Common/LoggedQsoManager.cs

@@ -0,0 +1,179 @@
+using System.Collections.Concurrent;
+using System.Text;
+using System.Text.Json;
+using WsjtxUtils.Searchlight.Common.Settings;
+using WsjtxUtils.WsjtxMessages.Messages;
+using WsjtxUtils.WsjtxMessages.QsoParsing;
+
+namespace WsjtxUtils.Searchlight.Common
+{
+    /// <summary>
+    /// Logged QSO manager behavior
+    /// </summary>
+    public enum QsoManagerBehavior
+    {
+        OncePerBand,
+        OncePerBandAndMode,
+    }
+
+    public class LoggedQsoManager
+    {
+        /// <summary>
+        /// Semaphore to manage log access
+        /// </summary>
+        private readonly SemaphoreSlim _qsoLogSemaphore = new SemaphoreSlim(1, 1);
+
+        /// <summary>
+        /// Settings that alter <see cref="nameof(LoggedQsoManager)"/> behavior
+        /// </summary>
+        private readonly LoggedQsoManagerSettings _settings;
+
+        /// <summary>
+        /// Platform specific newline characters
+        /// </summary>
+        private readonly ReadOnlyMemory<byte> _newlineBuffer;
+
+        /// <summary>
+        /// Logged QSO's key by band
+        /// </summary>
+        private readonly ConcurrentDictionary<ulong, ConcurrentBag<QsoLogged>> _loggedQso = new ConcurrentDictionary<ulong, ConcurrentBag<QsoLogged>>();
+
+        /// <summary>
+        /// 
+        /// </summary>
+        /// <param name="loggedQsoManagerSettings"></param>
+        public LoggedQsoManager(LoggedQsoManagerSettings loggedQsoManagerSettings)
+        {
+            _settings = loggedQsoManagerSettings;
+            _newlineBuffer = new ReadOnlyMemory<byte>(Encoding.UTF8.GetBytes(Environment.NewLine));
+        }
+
+        /// <summary>
+        /// Provides a list of previsouly contacted stations that are not highlighted already
+        /// </summary>
+        /// <param name="clientState"></param>
+        /// <returns></returns>
+        /// <exception cref="ArgumentNullException"></exception>
+        public IEnumerable<WsjtxQso> GetPreviouslyContactedStationsNotHiglighted(SearchlightClientState clientState)
+        {
+            if (clientState.Status == null)
+                throw new ArgumentNullException(nameof(clientState.Status));
+
+            string client = clientState.Id;
+            string callsign = clientState.Status.DECall;
+            string mode = clientState.Status.Mode;
+            ulong band = Utils.ApproximateBandFromFrequency(clientState.Status.DialFrequencyInHz);
+
+            return clientState.DecodedStations
+                .WherePreviouslyContacted(GetQsosLogged(band, callsign, mode))
+                .WhereNotPreviouslyHiglighted(clientState)
+                .Select(s => s.Value);
+        }
+
+        /// <summary>
+        /// Get all logged QSOs for the band and mode and specified callsign
+        /// </summary>
+        /// <param name="band"></param>
+        /// <param name="callsign"></param>
+        /// <param name="mode"></param>
+        /// <returns></returns>
+        public IEnumerable<QsoLogged> GetQsosLogged(ulong band, string callsign, string mode)
+        {
+            if (!_loggedQso.ContainsKey(band))
+                return new List<QsoLogged>();
+
+            var qsosForCallsign = _loggedQso[band].WhereDECallsign(callsign);
+
+            if (_settings.QsoManagerBehavior == QsoManagerBehavior.OncePerBand)
+                return qsosForCallsign;
+
+            return qsosForCallsign.WhereMode(mode);
+        }
+
+        /// <summary>
+        /// Log a <see cref="QsoLogged"/> message to the log file
+        /// </summary>
+        /// <param name="qsoLogged"></param>
+        /// <param name="cancellationToken"></param>
+        /// <returns></returns>
+        public async Task WriteQsoToLogFileAsync(QsoLogged qsoLogged, CancellationToken cancellationToken = default)
+        {
+            // log the message to the local cache
+            QsosLoggedForBand(qsoLogged.TXFrequencyInHz).Add(qsoLogged);
+
+            // check for a log file
+            if (string.IsNullOrEmpty(_settings.LogFilePath))
+                return;
+
+            // log the message to disk
+            await _qsoLogSemaphore.WaitAsync();
+            try
+            {
+                using(var stream = File.Open(_settings.LogFilePath, FileMode.Append))
+                {
+                    // write the object
+                    await JsonSerializer.SerializeAsync<QsoLogged>(stream,
+                        qsoLogged,
+                        new JsonSerializerOptions { WriteIndented = false },
+                        cancellationToken);
+
+                    // write a newline
+                    await stream.WriteAsync(_newlineBuffer, cancellationToken);
+                }
+            }
+            finally
+            {
+                _qsoLogSemaphore.Release();
+            }
+        }
+
+        /// <summary>
+        /// Load all logged QSOs from the log file
+        /// </summary>
+        /// <param name="func"></param>
+        /// <param name="cancellationToken"></param>
+        /// <returns></returns>
+        public async Task ReadAllQsosFromLogFileAsync(CancellationToken cancellationToken = default)
+        {
+            await _qsoLogSemaphore.WaitAsync();
+            try
+            {
+                using (TextReader r = new StreamReader(_settings.LogFilePath))
+                {
+                    string? line;
+                    while ((line = await r.ReadLineAsync()) != null)
+                    {
+                        QsoLogged ? qso = JsonSerializer.Deserialize<QsoLogged>(line);
+                        if (qso == null)
+                            continue;
+
+                        QsosLoggedForBand(qso.TXFrequencyInHz).Add(qso);
+                    }
+                }
+            }
+            finally
+            {
+                _qsoLogSemaphore.Release();
+            }
+        }
+
+        /// <summary>
+        /// Fetch the list of qso's for a given band
+        /// </summary>
+        /// <param name="frequencyInHertz"></param>
+        /// <returns></returns>
+        private ConcurrentBag<QsoLogged> QsosLoggedForBand(ulong frequencyInHertz)
+        {
+            var band = Utils.ApproximateBandFromFrequency(frequencyInHertz);
+
+            return _loggedQso.AddOrUpdate(band, (qsoBag) =>
+            {
+                return new ConcurrentBag<QsoLogged>();
+            },
+           (band, qsoBag) =>
+           {
+               return qsoBag;
+           });
+        }
+    }
+}

src/WsjtxUtils.Searchlight.Common/PskReporter/PskReporterUtils.cs

@@ -22,11 +22,10 @@ public static class PskReporterUtils
             Log.Debug("HTTP Get {url}.", url);
 
             var response = await httpClient.GetAsync(url, cancellationToken);
-            Log.Debug("Request complete: {url} {code} {headers}", url, response.StatusCode, response.Headers);
+            Log.Debug("Request complete: {url} {code}", url, response.StatusCode);
 
             response.EnsureSuccessStatusCode();
 
-            Log.Debug("Deserialize reception reports");
             var serializer = new XmlSerializer(typeof(PskReceptionReports));
             using var reader = new XmlTextReader(await response.Content.ReadAsStreamAsync(cancellationToken));
             return (PskReceptionReports?)serializer.Deserialize(reader);
@@ -46,11 +45,7 @@ private static HttpClient CreateHttpClientWithDecompressionSupport()
             if (handler.SupportsAutomaticDecompression)
                 handler.AutomaticDecompression = DecompressionMethods.All;
 
-            var client = new HttpClient(handler);
-
-            //client.DefaultRequestHeaders.UserAgent.Add(new System.Net.Http.Headers.ProductInfoHeaderValue("searchlight","0.1"));
-
-            return client;
+            return new HttpClient(handler);
         }
     }
 }

src/WsjtxUtils.Searchlight.Common/ReceptionReportState.cs

@@ -3,7 +3,7 @@
 namespace WsjtxUtils.Searchlight.Common
 {
     /// <summary>
-    /// Recpetion report state
+    /// Reception report state
     /// </summary>
     public class ReceptionReportState
     {
@@ -58,9 +58,24 @@ public ReceptionReportState(DateTime timestamp, PskReceptionReports? receptionRe
         /// </summary>
         public int Retry { get; set; }
 
+        /// <summary>
+        /// Time in seconds for exponential backoff
+        /// </summary>
+        public double Backoff { get; set; }
+
         /// <summary>
         /// Has the report been updated before a refresh?
         /// </summary>
         public bool IsDirty { get; set; }
+
+        /// <summary>
+        /// Handle exponential backoff on retry
+        /// </summary>
+        /// <param name="seconds"></param>
+        public void RetryWithExponentialBackoff(double seconds = 30)
+        {
+            Retry++;
+            Backoff = seconds * Math.Pow(2, Retry + Random.Shared.NextDouble()); // simple backoff with jitter
+        }
     }
 }