Merge pull request #211 from wildmountainfarms/dev

Dev

README.md

@@ -8,13 +8,15 @@
 
 Stores solar data in a database to view on Android, Grafana, or PVOutput
 
-<p style="text-align: center;">
-    <a href="#supported-products">Supported Products</a> &bull;
-    <a href="https://solarthing.readthedocs.io/">Documentation</a> &bull;
-    <a href="#features">Features</a> &bull;
-    <a href="#supported-databases">Supported Databases</a> &bull;
-    <a href="#examples">Examples</a>
-</p>
+View full documentation at https://solarthing.readthedocs.io/
+
+Contents
+
+* [Supported Products](#supported-products)
+* [Features](#features)
+* [Viewing Your Data](#viewing-your-data)
+* [Supported Databases](#supported-databases)
+* [Screenshots and Examples](#screenshots-and-examples)
 
 ## Supported Products
 * **Outback MATEs** (FX Inverter, MX/FM Charge Controller)
@@ -41,65 +43,56 @@ Ready to install? Use the [Quickstart](https://solarthing.readthedocs.io/en/late
 * Can [report CPU temperature](https://solarthing.readthedocs.io/en/latest/config/file/base-json/request/cpu-temperature.html).
 * Runs inside a Docker container
 
+
+## Viewing Your Data
+
+* Grafana (recommended)
+  * Use SolarThing Server (with CouchDB) alongside [Wild GraphQL Data Source](https://grafana.com/grafana/plugins/retrodaredevil-wildgraphql-datasource/)
+  * Alternatively, configure SolarThing to upload to InfluxDB for viewing of statistics in Grafana
+* [SolarThing Android](https://play.google.com/store/apps/details?id=me.retrodaredevil.solarthing.android)
+  * SolarThing Android connects directly to CouchDB to display data in a persistent notification
+* [PVOutput.org](https://pvoutput.org)
+  * Upload your data to CouchDB, then let SolarThing upload the data inside your database to PVOutput!
+
+If you are wondering how to set up SolarThing to view your data, you can head here: https://solarthing.readthedocs.io/en/latest/quickstart/data/index.html
+
 ## Supported Databases
+
 * CouchDB
-  * Allows for [SolarThing Android](https://github.com/wildmountainfarms/solarthing-android) and [SolarThing Server](https://solarthing.readthedocs.io/en/latest/quickstart/data/solarthing-server/index.html) to function
-  * Used for PVOutput data collection
-* GraphQL
-  * Allows use of CouchDB SolarThing data with Grafana
-  * Supplements the CouchDB database
+  * **Recommended** database and best supported database for SolarThing
+  * Used with [Wild GraphQL Data Source](https://grafana.com/grafana/plugins/retrodaredevil-wildgraphql-datasource/) to view data in Grafana
+  * Used with SolarThing Android to view data in the Android app
+  * Used as intermediate storage before data is aggregated and uploaded to PVOutput
 * InfluxDB
-  * Simplest to set up with Grafana
-* [PVOutput.org](https://pvoutput.org)
-  * Allows for viewing of data on [pvoutput.org](https://pvoutput.org)
-  * Requires CouchDB to be set up
+  * Upload statistics to InfluxDB and view them in your visualization tool of choice (Grafana is an option)
 * REST API
   * With the "post" database, all packets can be posted to a URL endpoint, useful for REST APIs
 
 
-## Examples
-PVOutput Wild Mountain Farms: [PVOutput System](https://pvoutput.org/intraday.jsp?sid=72206) and 
+## Screenshots and Examples
+
+You can get data in [Grafana](https://github.com/grafana/grafana) via **CouchDB+SolarThing Server** or via InfluxDB (InfluxDB not recommended).
+
+Grafana is customizable. Rearrange graphs and make it how you want!
+Pre-made Grafana dashboards are coming soon.
+
+![alt text](other/docs/grafana-screenshot-2024-04-21.png "SolarThing with Grafana")
+
+---
+
+PVOutput Wild Mountain Farms: [PVOutput System](https://pvoutput.org/intraday.jsp?sid=72206) and
 [PVOutput SolarThing Teams](https://pvoutput.org/listteam.jsp?tid=1528)
 
 ---
 
-SolarThing Android: [Github](https://github.com/wildmountainfarms/solarthing-android)
+SolarThing Android: [GitHub](https://github.com/wildmountainfarms/solarthing-android)
 |
 [Google Play](https://play.google.com/store/apps/details?id=me.retrodaredevil.solarthing.android)
 
 SolarThing Android displays data in a persistent notification that updates at a configurable rate
 ![alt text](other/docs/solarthing-android-notification-screenshot-1.jpg "SolarThing Android Notification")
 <hr/>
 
-You can get data in [Grafana](https://github.com/grafana/grafana) via InfluxDB or via CouchDB+SolarThing GraphQL.
-
-Grafana is very customizable. Rearrange graphs and make it how you want!
-![alt text](other/docs/grafana-screenshot-1.png "SolarThing with Grafana")
-
----
-
-### Usage at Wild Mountain Farms
-We monitor an Outback MATE2, Renogy Rover PG 40A, EPEver Tracer2210AN (20A) using a Raspberry Pi 3.
-Each device has its own instance of SolarThing running. Each instance uploads data to CouchDB. CouchDB, Grafana,
-and SolarThing GraphQL run on a separate "NAS" computer. This NAS runs the automation and pvoutput programs.
-The automation program handles the sending of Slack messages for low battery notifications.
-
-## Database Setup
-* [CouchDB setup](https://solarthing.readthedocs.io/en/latest/quickstart/config/database/couchdb.html)<br/>
-  * Used for SolarThing Android, and SolarThing Server
-* [InfluxDB 2.0 setup](https://solarthing.readthedocs.io/en/latest/quickstart/config/database/influxdb2.html)<br/>
-  * Used for direct Grafana queries
-
-## [Developer Use](other/docs/developer_use.md)
-## [Contributing](CONTRIBUTING.md)
-## [Updating](https://solarthing.readthedocs.io/en/latest/updating.html)
-
-## Configuration
-This uses all JSON for configuring everything. The files you edit are all in one place unless you decide to move them.
-
-See [configuration](https://solarthing.readthedocs.io/en/latest/configuration.html) to see how to set them up
-
-## [SolarThing Alternatives](https://solarthing.readthedocs.io/en/latest/misc/alternatives.html)
 
 ## Suggestions?
 If you have suggestions on how to improve the documentation or have a feature request, I'd love to

core/src/main/java/me/retrodaredevil/solarthing/packets/identification/IdentityInfo.java

@@ -19,7 +19,7 @@ public interface IdentityInfo {
 	// Note the JsonProperty annotations are for GraphQL. These are not meant to be serialized and saved to a database
 
 	@JsonProperty("displayName")
-	default String getDisplayName() { // FX 1, MX 2, Rover 40A
+	default @NotNull String getDisplayName() { // FX 1, MX 2, Rover 40A
 		String suffix = getSuffix();
 		if (suffix.isEmpty()) {
 			return getName();

server/src/main/java/me/retrodaredevil/solarthing/rest/graphql/PacketFinder.java

@@ -33,7 +33,7 @@ public PacketFinder(SimpleQueryHandler simpleQueryHandler) {
 		}
 	}
 	private void updateWithRange(long queryStart, long queryEnd) {
-		List<? extends InstancePacketGroup> rawPackets = simpleQueryHandler.queryStatus(queryStart, queryEnd, null);
+		List<? extends InstancePacketGroup> rawPackets = simpleQueryHandler.queryStatus(queryStart, queryEnd, null, null);
 		synchronized (cacheMap) {
 			for (InstancePacketGroup instancePacketGroup : rawPackets) {
 				int fragmentId = instancePacketGroup.getFragmentId();

server/src/main/java/me/retrodaredevil/solarthing/rest/graphql/SimpleQueryHandler.java

@@ -5,6 +5,7 @@
 import me.retrodaredevil.couchdbjava.CouchDbInstance;
 import me.retrodaredevil.solarthing.SolarThingConstants;
 import me.retrodaredevil.solarthing.annotations.NotNull;
+import me.retrodaredevil.solarthing.annotations.Nullable;
 import me.retrodaredevil.solarthing.config.databases.implementations.CouchDbDatabaseSettings;
 import me.retrodaredevil.solarthing.database.MillisDatabase;
 import me.retrodaredevil.solarthing.database.MillisQuery;
@@ -15,13 +16,17 @@
 import me.retrodaredevil.solarthing.database.couchdb.CouchDbSolarThingDatabase;
 import me.retrodaredevil.solarthing.database.exception.NotFoundSolarThingDatabaseException;
 import me.retrodaredevil.solarthing.database.exception.SolarThingDatabaseException;
+import me.retrodaredevil.solarthing.packets.collection.DefaultInstanceOptions;
+import me.retrodaredevil.solarthing.packets.collection.FragmentedPacketGroup;
+import me.retrodaredevil.solarthing.packets.collection.InstancePacketGroup;
+import me.retrodaredevil.solarthing.packets.collection.PacketGroup;
+import me.retrodaredevil.solarthing.packets.collection.PacketGroups;
+import me.retrodaredevil.solarthing.packets.collection.parsing.PacketParsingErrorHandler;
+import me.retrodaredevil.solarthing.rest.exceptions.DatabaseException;
 import me.retrodaredevil.solarthing.type.alter.StoredAlterPacket;
 import me.retrodaredevil.solarthing.type.closed.meta.DefaultMetaDatabase;
 import me.retrodaredevil.solarthing.type.closed.meta.EmptyMetaDatabase;
 import me.retrodaredevil.solarthing.type.closed.meta.MetaDatabase;
-import me.retrodaredevil.solarthing.packets.collection.*;
-import me.retrodaredevil.solarthing.packets.collection.parsing.PacketParsingErrorHandler;
-import me.retrodaredevil.solarthing.rest.exceptions.DatabaseException;
 import me.retrodaredevil.solarthing.type.closed.meta.RootMetaPacket;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -31,7 +36,6 @@
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
-import java.util.NoSuchElementException;
 import java.util.Objects;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Future;
@@ -89,7 +93,7 @@ public List<? extends FragmentedPacketGroup> sortPackets(List<? extends Instance
 	 * @param sourceId The source ID or null. If null, the returned List may contain packet groups from multiple sources
 	 * @return The resulting packets
 	 */
-	private List<? extends InstancePacketGroup> queryPackets(MillisDatabase database, long from, long to, String sourceId) {
+	private List<? extends InstancePacketGroup> queryPackets(MillisDatabase database, long from, long to, @Nullable String sourceId, @Nullable Integer fragmentId) {
 
 		MillisQuery millisQuery = new MillisQueryBuilder()
 				.startKey(from)
@@ -148,21 +152,21 @@ private List<? extends InstancePacketGroup> queryPackets(MillisDatabase database
 			}
 			return Collections.emptyList();
 		}
-		if (sourceId == null) {
-			return PacketGroups.parseToInstancePacketGroups(rawPacketGroups, defaultInstanceOptions);
-		}
-		Map<String, List<InstancePacketGroup>> map = PacketGroups.parsePackets(rawPacketGroups, defaultInstanceOptions);
-		if(map.containsKey(sourceId)){
-			List<InstancePacketGroup> instancePacketGroupList = map.get(sourceId);
-			return PacketGroups.orderByFragment(instancePacketGroupList);
-		}
-		throw new NoSuchElementException("No element with sourceId: '" + sourceId + "' available keys are: " + map.keySet());
+		// Note: Before 2024-04-05 this method would throw a NoSuchElementException if no packets were found under a given Source ID
+		//   Additionally PacketGroups.orderByFragment() was used to order the result ONLY IF a Source ID was provided.
+		//   This behavior of this method has changed since then, which may have unintended effects.
+		//   I really don't know if the ordering by Fragment ID was necessary, and I also don't think we really need that exception to be thrown.
+		return rawPacketGroups.stream()
+				.map(packetGroup -> PacketGroups.parseToInstancePacketGroup(packetGroup, defaultInstanceOptions))
+				.filter(instancePacketGroup -> sourceId == null || instancePacketGroup.getSourceId().equals(sourceId))
+				.filter(instancePacketGroup -> fragmentId == null || instancePacketGroup.getFragmentId() == fragmentId)
+				.toList();
 	}
-	public List<? extends InstancePacketGroup> queryStatus(long from, long to, String sourceId) {
-		return queryPackets(database.getStatusDatabase(), from, to, sourceId);
+	public List<? extends InstancePacketGroup> queryStatus(long from, long to, @Nullable String sourceId, @Nullable Integer fragmentId) {
+		return queryPackets(database.getStatusDatabase(), from, to, sourceId, fragmentId);
 	}
-	public List<? extends InstancePacketGroup> queryEvent(long from, long to, String sourceId) {
-		return queryPackets(database.getEventDatabase(), from, to, sourceId);
+	public List<? extends InstancePacketGroup> queryEvent(long from, long to, @Nullable String sourceId, @Nullable Integer fragmentId) {
+		return queryPackets(database.getEventDatabase(), from, to, sourceId, fragmentId);
 	}
 
 	public MetaDatabase queryMeta() {

server/src/main/java/me/retrodaredevil/solarthing/rest/graphql/packets/FragmentFilter.java

@@ -2,6 +2,10 @@
 
 import me.retrodaredevil.solarthing.rest.graphql.packets.nodes.PacketNode;
 
+/**
+ * @deprecated Should not be needed anymore as {@link me.retrodaredevil.solarthing.rest.graphql.SimpleQueryHandler} accepts fragmentIds for most of its methods now.
+ */
+@Deprecated
 public class FragmentFilter implements PacketFilter {
 	private final int fragmentId;
 

server/src/main/java/me/retrodaredevil/solarthing/rest/graphql/service/SchemaConstants.java

@@ -11,6 +11,7 @@ public final class SchemaConstants {
 	public static final String DESCRIPTION_TO = "The maximum time in milliseconds since the epoch to get data from.";
 	public static final String DESCRIPTION_OPTIONAL_SOURCE = "The Source ID to include packets from, or null to include packets from multiple sources.";
 	public static final String DESCRIPTION_REQUIRED_SOURCE = "The Source ID to include packets from.";
-	public static final String DESCRIPTION_FRAGMENT_ID = "The fragment ID to include data from.";
+	public static final String DESCRIPTION_OPTIONAL_FRAGMENT_ID = "The fragment ID to include data from.";
+	public static final String DESCRIPTION_REQUIRED_FRAGMENT_ID = "The fragment ID to include data from.";
 
 }

server/src/main/java/me/retrodaredevil/solarthing/rest/graphql/service/SolarThingGraphQLDailyService.java

@@ -280,6 +280,7 @@ public SolarThingFullDayStatusQuery queryFullDay(
 			@GraphQLArgument(name = "from", description = "The epoch millis value that will be used to determine the starting day. Set to null to guarantee a query of a single day.") @Nullable Long from,
 			@GraphQLArgument(name = "to", description = "The epoch millis value that will be used to determine the ending day.") long to,
 			@GraphQLArgument(name = "sourceId", description = DESCRIPTION_OPTIONAL_SOURCE) @Nullable String sourceId,
+			@GraphQLArgument(name = "fragmentId", description = DESCRIPTION_OPTIONAL_FRAGMENT_ID) @Nullable Integer fragmentId,
 			@GraphQLArgument(name = "useCache", defaultValue = "false") boolean useCache){
 
 		LocalDate fromDate = Instant.ofEpochMilli(from == null ? to : from).atZone(zoneId).toLocalDate();
@@ -292,7 +293,8 @@ public SolarThingFullDayStatusQuery queryFullDay(
 			List<? extends InstancePacketGroup> packets = simpleQueryHandler.queryStatus(
 					queryStart,
 					queryEnd,
-					sourceId
+					sourceId,
+					fragmentId
 			);
 			return new SimpleSolarThingFullDayStatusQuery(new BasicPacketGetter(packets, PacketFilter.KEEP_ALL), simpleQueryHandler.sortPackets(packets, sourceId));
 		}

server/src/main/java/me/retrodaredevil/solarthing/rest/graphql/service/SolarThingGraphQLFXService.java

@@ -53,7 +53,8 @@ public List<DataNode<FXChargingPacket>> queryFXCharging(
 		}
 
 		long startTime = from - 3 * 60 * 60 * 1000; // 3 hours back
-		List<? extends InstancePacketGroup> packets = simpleQueryHandler.queryStatus(startTime, to, null);
+		// Don't filter on fragmentId here. Even though we are provided with one, we still want data from other fragments as those might have temperature sensor data
+		List<? extends InstancePacketGroup> packets = simpleQueryHandler.queryStatus(startTime, to, null, null);
 
 		// We make masterIdIgnoreDistance null because we will only be using fragmentId as the master fragment ID
 		Map<String, List<FragmentedPacketGroup>> map = PacketGroups.sortPackets( // separate based on source ID