Update spec with wafer map API decision

change/@ni-nimble-components-c0f72f9f-b879-43ad-bc5b-ee8b74e2700f.json

@@ -0,0 +1,7 @@
+{
+  "type": "none",
+  "comment": "Update rendering spec for Wafer Map component with API changes",
+  "packageName": "@ni/nimble-components",
+  "email": "33986780+munteannatan@users.noreply.github.com",
+  "dependentChangeType": "none"
+}

packages/nimble-components/src/wafer-map/specs/features/rendering.md

@@ -82,46 +82,50 @@ The POC is found in this branch [Worker Rendering POC](https://github.com/ni/nim
 
 ### Data Structure and Interface
 
-We have two possible solutions for representing the data in the memory. They will be decided with a spec update. The fist one is an in-house solution:
+The best solution to solve the API of the wafermap is to use Apache Arrow as the wafer component API, and Typed Arrays as teh worker API for their iterating performance and transferability to worker threads.
+
+The Public API will be the following:
 
 ```TS
-class WaferData {
-    // the x coordinates of each column of dies
-    dieColIndexArray: Int32Array;
-    // the lengths of each row of dies
-    rowLengthsArray: Int32Array;
-    // the y coordinates of each die as a matrix row by row
-    dieRowIndexLayer: Int32Array;
-    // the value of each die as a matrix row by row
-    dieValuesLayer: Float64Array;
-    // the highlight  approach is still undecided, we have two options:
-    // the highlight state of each die as a matrix; user will have to pre-calculate tags into highlighted conditions.
-    dieHighlightsLayer: Int8Array;
-    // a 32 bitset array of tags for each die; aligns more closely to the existing public api but limits users to 32 tags.
-    dieHighlightsLayer: Int32Array;
-    // metadata array for each die; it will not be sent to the worker
-    metadata : unknown[]
+import { Table } from 'apache-arrow';
+export class WaferMap extends FoundationElement {
+...
+public diesTable: Table<{
+        colIndex: Int32,
+        rowIndex: Int32,
+        value: Float32,
+        tags: Uint32;
+        metadata: never;
+    }>
+...
 }
 ```
 
-Using TypedArrays has the benefit of direct transfer to web workers without structured cloning of the object by transferring the arrayBuffers and reconstructing the object. Other benefits of typedArrays include the low access time when iterating over the values, more memory efficiency and faster direct access to metadata layers values. The previous inputs can be adapted to this new structure to maintain backwards compatibility.
+This will be the [Apache Arrow](https://arrow.apache.org/docs/js/classes/Arrow_dom.Table.html) table schema.
+The row and column indices will be `Int32` columns, the values will be `Float32` columns.
+The tags for each die will be represented as a 32 bit mask stored in a `Uint32` column.
+The metadata column will be stored in an wildcard typed column.
 
-This API will have [optimized byte-array interop from Blazor](https://learn.microsoft.com/en-us/dotnet/core/compatibility/aspnet-core/6.0/byte-array-interop) and should be supported by Angular as a [vanilla javascript feature](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer).
+This approach has the benefits of a row based format that aligns well with the existing public API, as well as a nice public API that easily allows future improvements.
 
-The alternative to the above mentioned data structure is an [apache arrow](https://arrow.apache.org/docs/js/index.html) table with columns and metadata.
+The limits for this approach are the following:
 
-Pros of using Apache Arrow:
-
--   A row based format that aligns well with the existing public api
--   Well supported and tested format
--   Nice public API to use, we don't have to invent a new format, just document our schema for the arrow tables
--   Designed for large dataset visualizations
+1. There seems to be no support for columns of lists of strings. We decided to overcome this using a bit mask of tags. Another possible solution can be a dynamic number of columns for storing tags, but the performance may suffer.
+2. There is no support currently for [searching or filtering the table](https://github.com/apache/arrow/issues/13233). The possible solutions for this are searching by iterating over the whole table, which is not feasible (see 4.) or using a higher level library such as [aquero](https://uwdata.github.io/arquero/). Searching for dies based on their position is crucial for highlighting and sending the highlighted die metadata with the `die-hover` event. The solution we chose is using a custom method for finding rows based on column and row indexes cached as typed arrays. This method provides faster access to row values and metadata and does not induce additional dependencies.
+3. The transfer method between the main an worker thread for arrow tables is cumbersome, we would have to use another higher level library [geoarrow](https://github.com/geoarrow/geoarrow-js/blob/main/src/worker/transferable.ts). Fortunately we can skip over this problem by not transferring the tables to the worker.
+4. The iteration over stored rows is very slow compared to typed arrays as seen in the table below. This impacts the goals we set for this rendering improvement. The solution to this issue and the transferring issue is splitting the relevant columns from the table (rows, columns, values, tags mask) and messaging them to the worker separately. This can be done with a very small overhead using the [getChild](https://arrow.apache.org/docs/js/classes/Arrow_dom.Table.html#getChild) method and calling [toArray](https://arrow.apache.org/docs/js/classes/Arrow_dom.Vector.html#toArray) on the resulting vector. After being transferred, The buffers can be cached to speed up value access and filtering.
 
-In order to choose from these alternatives we will prototype and check:
+| name                      | duration (ms) [1]  | duration (ms) [2]  | detail                                                                                                    |
+| ------------------------- | ------------------ | ------------------ | --------------------------------------------------------------------------------------------------------- |
+| typed iterate             | 7.551699995994568  | 6.052600026130676  | iterating over two 1M typed arrays and calculating the sums                                               |
+| typed from table iterate  | 6.4953999519348145 | 5.136900067329407  | iterating over two 1M typed arrays from Table columns and calculating the sums (time includes conversion) |
+| vector iterate            | 76.4708000421524   | 66.58230006694794  | iterating over two 1M Vectors and calculating the sums                                                    |
+| table get() iterate       | 1350.0404000282288 | 1030.582899928093  | iterating over the 1M Table using `table.get(rowIndex)` and calculating the sums                          |
+| table [iterator] iterate  | 1091.6706000566483 | 1011.069100022316  | iterating over the 1M Table using the [iterator] and calculating the sums                                 |
+| array from table iterate  | 943.0076999664307  | 980.0875999927521  | iterating over the 1M Table after converting `toArray()` and calculating the sums                         |
+| vector from table iterate | 965.2465000152588  | 1012.9023000001907 | iterating over the 1M Vector after converting the Table with `makeVector()` and calculating the sums      |
 
--   Does it have comparable memory performance
--   Does it perform well or have significant overhead
--   Is it easy to divide and use in parallel
+The memory impact is not very significant, amounting to 74.01MB for 1M dies compared with 44.65MB for the previously prototyped API.
 
 ### Rendering
 
@@ -210,6 +214,39 @@ The current expectation is for a singular wafer component to be displayed on the
 
 ### Alternative Data Structures and Interfaces
 
+The alternative to using Apache Arrow tables is an in-house solution:
+
+```TS
+class WaferData {
+    // the x coordinates of each column of dies
+    dieColIndexArray: Int32Array;
+    // the lengths of each row of dies
+    rowLengthsArray: Int32Array;
+    // the y coordinates of each die as a matrix row by row
+    dieRowIndexLayer: Int32Array;
+    // the value of each die as a matrix row by row
+    dieValuesLayer: Float64Array;
+    // the highlight  approach is still undecided, we have two options:
+    // the highlight state of each die as a matrix; user will have to pre-calculate tags into highlighted conditions.
+    dieHighlightsLayer: Int8Array;
+    // a 32 bitset array of tags for each die; aligns more closely to the existing public api but limits users to 32 tags.
+    dieHighlightsLayer: Int32Array;
+    // metadata array for each die; it will not be sent to the worker
+    metadata : unknown[]
+}
+```
+
+Using TypedArrays has the benefit of direct transfer to web workers without structured cloning of the object by transferring the arrayBuffers and reconstructing the object. Other benefits of typedArrays include the low access time when iterating over the values, more memory efficiency and faster direct access to metadata layers values. The previous inputs can be adapted to this new structure to maintain backwards compatibility.
+
+This API will have [optimized byte-array interop from Blazor](https://learn.microsoft.com/en-us/dotnet/core/compatibility/aspnet-core/6.0/byte-array-interop) and should be supported by Angular as a [vanilla javascript feature](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer).
+
+Pros of using Apache Arrow:
+
+-   A row based format that aligns well with the existing public api
+-   Well supported and tested format
+-   Nice public API to use, we don't have to invent a new format, just document our schema for the arrow tables
+-   Designed for large dataset visualizations
+
 Another option is to break each object property as a separate attribute for the wafer map component. This can also lead to increased complexity and confusion for the user which will need to pass several structured objects instead of a singular object.
 
 ### Alternative Rendering
@@ -235,6 +272,7 @@ User Indication for [interactions in progress (>200ms)](https://web.dev/articles
 -   the wafer-map will use bitmap scaling in addition to a spinner
 -   the wafer-map will immediately show the spinner / fire event or only after, for example 200ms
 -   the renderer will report progress for larger wait times.
+-   the rendering will be done sequentially in animation frames so the user will see the progress at 60Hz
 
 A follow-on HLD update will specify the approved decision.