9.1.0 release roundup

documentation/clients/java_ilp.md

@@ -168,6 +168,49 @@ There are three ways to create a client instance:
        // ...
    }
    ```
+
+## Configuring multiple urls
+
+:::note
+
+This feature requires QuestDB OSS 9.1.0+ or Enterprise 3.0.4+.
+
+:::
+
+The ILP client can be configured with multiple _possible_ endpoints to send your data to. Only one will be sent to at
+any one time.
+
+To configure this feature, simply provide multiple `addr` entries. For example:
+
+
+```java
+try (Sender sender = Sender.fromConfig("http::addr=localhost:9000;addr=localhost:9999;")) {
+   // ...
+}
+```
+
+On initialisation, if `protocol_version=auto`, the sender will identify the first instance that is writeable. Then it will _stick_ to this instance and write
+any subsequent data to it.
+
+In the event that the instance becomes unavailable for writes, the client will retry the other possible endpoints, and when it finds 
+a new writeable instance, will _stick_ to it instead. This unvailability is characterised by failures to connect or locate the instance, 
+or the instance returning an error code due to it being read-only.
+
+By configuring multiple addresses, you can continue allowing you to continue to capture data if your primary instance
+fails, without having to reconfigure the clients. This backup instance can be hot or cold, and so long as it is assigned a known address, it will be written to as soon as it is started.
+
+Enterprise users can leverage this feature to transparently handle replication failover, without the need to introduce a load-balancer or
+reconfigure clients.
+
+:::tip
+
+You may wish to increase the value of `retry_timeout` if you expect your backup instance to take a large amount of time to become writeable.
+
+For example, when performing a primary migration (Enterprise replication), with default settings, you might want to increase this
+to `30s` or higher.
+
+:::
+
 
 ## General usage pattern
 
@@ -289,6 +332,13 @@ closing the client.
 
 ## Error handling
 
+
+:::note
+
+If you have configured multiple addresses, retries will be run against different instances.
+
+:::
+
 HTTP automatically retries failed, recoverable requests: network errors, some
 server errors, and timeouts. Non-recoverable errors include invalid data,
 authentication errors, and other client-side errors.
@@ -318,6 +368,17 @@ With TCP transport, you don't have this option. If you get an exception, you
 can't continue with the same client instance, and don't have insight into which
 rows were accepted by the server.
 
+:::caution
+
+Error handling behaviour changed with the release of QuestDB 9.1.0.
+
+Previously, failing all retries would cause the code to except and release the buffered data.
+
+Now the buffer will not be released. If you wish to re-use the same sender with fresh data, you must call the
+new `reset()` function.
+
+:::
+
 ## Designated timestamp considerations
 
 The concept of [designated timestamp](/docs/concept/designated-timestamp/) is

documentation/concept/sql-optimizer-hints.md

@@ -28,28 +28,25 @@ Hints are designed to be a safe optimization mechanism:
 
 -----
 
-## Binary Search Optimizations and Hints
+## Time-series JOIN hints
 
 Since QuestDB 9.0.0, QuestDB's optimizer defaults to using a binary search-based strategy for **`ASOF JOIN`** and
 **`LT JOIN`** (Less Than Join) queries that have a filter on the right-hand side (the joined or lookup table). This
 approach is generally faster as it avoids a full table scan.
 
 However, for some specific data distributions and filter conditions, the previous strategy of performing a parallel full
-table scan can be more performant. For these cases, QuestDB provides hints to *avoid* the default binary search.
+table scan can be more performant. For these cases, QuestDB provides hints to modify the default search strategy.
 
-### AVOID\_ASOF\_BINARY\_SEARCH and AVOID\_LT\_BINARY\_SEARCH
+The `asof`-prefixed hints will also apply to `lt` joins.
 
-These hints instruct the optimizer to revert to the pre-9.0 execution strategy for `ASOF JOIN` and `LT JOIN` queries,
+### `asof_linear_search(l r)`
+
+This hint instructs the optimizer to revert to the pre-9.0 execution strategy for `ASOF JOIN` and `LT JOIN` queries,
 respectively. This older strategy involves performing a full parallel scan on the joined table to apply filters *before*
 executing the join.
 
-- `AVOID_ASOF_BINARY_SEARCH(left_table_alias right_table_alias)`: Use for **`ASOF JOIN`** queries.
-- `AVOID_LT_BINARY_SEARCH(table_alias)`: Use for **`LT JOIN`** queries.
-
-<!-- end list -->
-
-```questdb-sql title="Avoiding binary search for an ASOF join"
-SELECT /*+ AVOID_ASOF_BINARY_SEARCH(orders md) */
+```questdb-sql title="Using linear search for an ASOF join"
+SELECT /*+ asof_linear_search(orders md) */
   orders.ts, orders.price, md.md_ts, md.bid, md.ask
 FROM orders
 ASOF JOIN (
@@ -68,20 +65,20 @@ The **default strategy (binary search)** works as follows:
    evaluating the filter condition until a match is found.
 
 <Screenshot
-alt="Diagram showing execution of the USE_ASOF_BINARY_SEARCH hint"
+alt="Diagram showing execution of the asof_linear_search hint"
 height={447}
 src="images/docs/concepts/asof-join-binary-search-strategy.svg"
 width={745}
 />
 
-The **hinted strategy (`AVOID_..._BINARY_SEARCH`)** forces this plan:
+The hinted strategy forces this plan:
 
 1. Apply the filter to the *entire* joined table in parallel.
 2. Join the filtered (and now much smaller) result set to the main table.
 
-#### When to use the AVOID hints
+#### When to use it
 
-You should only need these hints in a specific scenario: when the filter on your joined table is **highly selective**.
+You should only need this hint in a specific scenario: when the filter on your joined table is **highly selective**.
 
 A filter is considered highly selective if it eliminates a very large percentage of rows (e.g., more than 95%). In this
 situation, the hinted strategy can be faster because:
@@ -95,6 +92,52 @@ scan may have to check many rows before finding one that satisfies the filter co
 For most other cases, especially with filters that have low selectivity or when the joined table data is not in
 memory ("cold"), the default binary search is significantly faster as it minimizes I/O operations.
 
+### `asof_index_search(l r)`
+
+This hint instructs the optimizer to use a symbol's index to skip over any time partitions where the symbol does not appear. 
+
+In partitions where the symbol does appear, there will still be some scanning to locate the matching rows.
+
+```questdb-sql title="Using index search for an ASOF join"
+SELECT /*+ asof_index_search(orders md) */
+    orders.timestamp, orders.symbol, orders.price
+FROM orders
+ASOF JOIN (md) ON (symbol);
+```
+
+#### When to use it
+
+When your symbol column has a highly selective index i.e. the symbol entry is rare, rarely appearing in any of
+your partitions. 
+
+If the symbol appears frequently, then this hint may cause a slower execution plan than the default.
+
+If no index exists on the column, this hint will be disregarded.
+
+### `asof_memoized_search(l r)`
+
+This hint instructs the optimizer to memoize (remember) rows it has previously seen, and use this information to avoid 
+repeated re-scanning of data.
+
+Imagine a linear scan. For each symbol, we must scan forward to find the next available row. This symbol could be far away.
+When the matching row is located, we store it, pick the next symbol, and repeat this scan. This causes repeated re-reading of data.
+
+Instead, the query engine will check each row for a matching symbol, recording the locations. Then when the symbol is next
+processed, the memoized rows are checked (look-ahead) and the cursor skips forward.
+
+```questdb-sql title="Using memoized search for an ASOF join"
+SELECT /*+ asof_memoized_search(orders md) */
+    orders.timestamp, orders.symbol, orders.price
+FROM orders
+ASOF JOIN (md) ON (symbol);
+```
+
+#### When to use it
+
+If your table has a very skewed symbol distribution, this hint can dramatically speed up the query. A typical skew
+would be a few symbols with very large row counts, and many symbols with very small row counts. This hint works well
+for Zipfian-distributed data.
+
 -----
 
 ### Execution Plan Observation
@@ -133,10 +176,10 @@ SelectedRecord
 
 #### Hinted Execution Plan (Full Scan)
 
-When you use the `AVOID_ASOF_BINARY_SEARCH` hint, the plan changes.
+When you use the `asof_linear_search` hint, the plan changes.
 
 ```questdb-sql title="Observing execution plan with the AVOID hint" demo
-EXPLAIN SELECT /*+ AVOID_ASOF_BINARY_SEARCH(core_price market_data) */
+EXPLAIN SELECT /*+ asof_linear_search(core_price market_data) */
   *
 FROM core_price
 ASOF JOIN market_data
@@ -161,3 +204,9 @@ SelectedRecord
                 Frame forward scan on: market_data
 ```
 
+## Deprecated hints
+
+- `avoid_asof_binary_search`
+  - superceded by `asof_linear_search`
+- `avoid_lt_binary_search`
+  - superceded by `asof_linear_search`

documentation/configuration-utils/_cairo.config.json

@@ -291,6 +291,10 @@
     "default": "false",
     "description": "Sets debug flag for JIT compilation. When enabled, assembly will be printed into `stdout`."
   },
+  "cairo.sql.jit.max.in.list.size.threshold": {
+    "default": "10",
+    "description": "Controls whether or not JIT compilation will be used for a query that uses the IN predicate. If the IN list is longer than this threshold, JIT compilation will be cancelled."
+  },
   "cairo.sql.jit.bind.vars.memory.page.size": {
     "default": "4K",
     "description": "Sets the memory page size for storing bind variable values for JIT compiled filter."
@@ -454,5 +458,13 @@
   "cairo.system.writer.data.append.page.size": {
     "default": "256k",
     "description": "mmap sliding page size that TableWriter uses to append data for each column specifically for System tables."
+  },
+  "cairo.file.descriptor.cache.enabled": {
+    "default": "true",
+    "description": "enables or disables the file-descriptor cache"
+  },
+  "cairo.partition.encoder.parquet.raw.array.encoding.enabled": {
+    "default": "false",
+    "description": "determines whether to export arrays in QuestDB-native binary format (true, less compatible) or Parquet-native format (false, more compatible)."
   }
 }

documentation/configuration-utils/_replication.config.json

@@ -43,6 +43,50 @@
     "default": "1000",
     "description": "Millisecond polling rate of a replica instance to check for the availability of new changes."
   },
+  "replication.primary.sequencer.part.txn.count": {
+    "default": "5000",
+    "description": "Sets the txn chunking size for each compressed batch. Smaller is better for constrained networks (but more costly)."
+  },
+  "replication.primary.checksum=service-dependent": {
+    "default": "service-dependent",
+    "description": "Where a checksum should be calculated for each uploaded artifact. Required for some object stores. Other options: never, always"
+  },
+  "replication.primary.upload.truncated": {
+    "default": "true",
+    "description": "Skip trailing, empty column data inside a WAL column file."
+  },
+  "replication.requests.buffer.size": {
+    "default": "32768",
+    "description": "Buffer size used for object-storage downloads."
+  },
+  "replication.summary.interval": {
+    "default": "1m",
+    "description": "Frequency for printing replication progress summary in the logs."
+  },
+  "replication.metrics.per.table": {
+    "default": "true",
+    "description": "Enable per-table replication metrics on the prometheus metrics endpoint."
+  },
+  "replication.metrics.dropped.table.poll.count": {
+    "default": "10",
+    "description": "How many scrapes of prometheus metrics endpoint before dropped tables will no longer appear."
+  },
+  "replication.requests.max.batch.size.fast": {
+    "default": "64",
+    "description": "Number of parallel requests allowed during the 'fast' process (non-resource constrained)."
+  },
+  "replication.requests.max.batch.size.slow": {
+    "default": "2",
+    "description": "Number of parallel requests allowed during the 'slow' process (error/resource constrained path)."
+  },
+  "replication.requests.base.timeout": {
+    "default": "10s",
+    "description": "Replication upload/download request timeout."
+  },
+  "replication.requests.min.throughput": {
+    "default": "262144",
+    "description": "Expected minimum network speed for replication transfers. Used to expand the timeout and account for network delays."
+  },
   "native.async.io.threads": {
     "default": "cpuCount",
     "description": "The number of async (network) io threads used for replication (and in the future cold storage). The default should be appropriate for most use cases."

documentation/deployment/systemd.md

@@ -141,3 +141,29 @@ is activated or de-activated. You also do not need to apply `sudo` to make
 changes to the services.
 
 Consistent with the examples on this page, we recommend scoped users.
+
+
+## Daily timers
+
+If running QuestDB on a `systemd` based Linux (for example, `Ubuntu`) you may find that, by default, there are a number of daily upgrade timers enabled. 
+
+When executed, these tasks restart `systemd` services, which can cause interruptions to QuestDB. It will appear
+that QuestDB restarted with no errors or apparent trigger.
+
+To resolve it, either:
+
+- Force services to be listed for restart, but not restarted automatically.
+  - Modify `/etc/needrestart/needrestart.conf` to contain `$nrconf{restart} = 'l'`.
+- Disable the auto-upgrade services entirely:
+
+```bash
+sudo systemctl disable --now apt-daily-upgrade.timer
+sudo systemctl disable --now apt-daily.timer
+sudo systemctl disable --now unattended-upgrades.service
+```
+
+You can check the status of the timers using:
+
+```bash
+systemctl list-timers --all | grep apt
+```