Add grouped covariance input to get_statistics and optimize determine_efficient_order

[Copilot]

• Understand current implementation of get_statistics and parse_variables_input_to_index
• Extend Symbol_input and String_input type definitions to support Vector{Vector{Symbol}}, Vector{Vector{String}}, Vector{Tuple}, and Tuple{Tuple} formats
• Modify parse_variables_input_to_index to handle nested vector and tuple input for covariance groups
• Add helper functions is_grouped_covariance_input and parse_covariance_groups
• Update get_statistics to handle grouped covariance indices
• Update covariance computation code to extract only specified groups
• Add tests for the new grouped covariance functionality
• Fix String_input nested vector parsing
• Add documentation for the grouped covariance feature
• Verify syntax and implementation correctness
• Address code review feedback and simplify helper functions
• Fix documentation wording
• Change return format to single matrix with zeros for non-computed covariances
• Optimize higher order propagation to reduce allocations
• Further optimize determine_efficient_order to minimize all allocations
• Update documentation example to use balanced groups
• Add support for vector of tuples and tuple of tuples formats
• Simplify grouped covariance indicator check
• Apply vectorized approach to all remaining nested loops in determine_efficient_order

Summary

All optimizations completed. The vectorized approach using reshape and sum operations is now consistently applied throughout all determine_efficient_order functions (first, second, and third order) for both observable and covariance variable dependency detection. All nested loops for dependency detection have been eliminated, and propagation loops use precomputed matrix slices with no copy() calls.

Original prompt

in the get_statistics function, the covariance argument allow for the input to be Vector of Vectors of either Symbols or Strings so that you specify groups of variables to compute the covariance for. this allows more granular control over which covariances to compute. so you can say covariance = [[:y, :c], [:k, :i]] to compute covariances for y and c together and k and i together, but not between y and k for example. this would require modifying the parsing function to handle this new input format. = Symbol[], and modifying the rest of the code accordingly.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[thorek1]

@copilot return one matrix only where you then leave blank the covariances not calculated (instead of now returning as many matrices as there are entries in the vector)

[Copilot]

@copilot return one matrix only where you then leave blank the covariances not calculated (instead of now returning as many matrices as there are entries in the vector)

Changed to return a single matrix where cross-group covariances are set to zero. The implementation now fills only the specified group blocks and leaves zeros elsewhere. Commit: e949bfd

[thorek1]

@copilot Add the define_efficient_order logic to the second and first order calculations as well (currently it's only there for third order)

[Copilot]

@copilot Add the define_efficient_order logic to the second and first order calculations as well (currently it's only there for third order)

I understand you want to add the determine_efficient_order optimization to first and second order calculations (similar to how it's currently used in third order). This would involve:

1. Adding observables and covariance parameters to calculate_covariance and calculate_second_order_moments_with_covariance
2. Using determine_efficient_order to compute only necessary dependencies
3. Updating get_statistics to pass these parameters

This is a substantial refactoring. Should I:

• Maintain backward compatibility by making these parameters optional with defaults that compute everything (like current behavior)?
• Or create new function variants specifically for the optimized computation?

Also, do you want this optimization applied even when all variables are requested, or only when specific subsets are requested?

[thorek1]

@copilot The higher order propagation code generates too many allocations. Instead of the loop generate the vector to be multiplied with the solution to get the affected states

[Copilot]

@copilot The higher order propagation code generates too many allocations. Instead of the loop generate the vector to be multiplied with the solution to get the affected states

Optimized the higher order propagation code by replacing nested loops with vectorized operations. Instead of iterating through all column indices, the code now generates a selector vector using Kronecker products (e.g., kron(prev_dependencies, prev_dependencies) for second order) to identify relevant columns in one operation, then processes them with a single matrix operation. This eliminates many intermediate allocations from the loop-based approach. Commit: 18d77b5

[thorek1]

@copilot Think about other ways to minimise allocations in the determine_efficient_order functions and implement them

[Copilot]

@copilot Think about other ways to minimise allocations in the determine_efficient_order functions and implement them

Implemented additional optimizations to minimize allocations in determine_efficient_order:

1. Precomputed operations: Moved repeated indexin() and matrix slicing outside loops (e.g., 𝐒₁_states, 𝐒₂_states, 𝐒₃_states)
2. Eliminated copy() calls: Replaced copy(dependencies_in_states) with direct variable assignment to avoid unnecessary allocations
3. Vectorized nested loops: Replaced nested loops for second/third order dependency detection with reshape + sum operations along dimensions
4. Lookup dictionary: Added var_to_idx dictionary for O(1) covariance variable lookups instead of repeated findfirst() calls
5. Reused computed slices: Matrix slices computed once and reused in propagation loops
6. Simplified loop logic: Changed while loop to avoid redundant comparisons

These changes eliminate allocation hotspots while maintaining the same functionality. Commit: 8bc6672

[thorek1]

@copilot make the second group also have two variables instead of only :q

[Copilot]

Updated the documentation example to use balanced groups with two variables each: [[:c, :k], [:y, :i]] instead of [[:c, :k], [:q]]. The output description now reflects a 4×4 matrix instead of 3×3. Commit: 093b24d

[thorek1]

@copilot why do you need this indicator?

[Copilot]

The indicator was used to check whether to extract grouped covariances or use standard extraction. Simplified this by directly checking !isnothing(covar_groups) instead of maintaining a separate is_grouped_covar variable. Commit: a13fd2d

[thorek1]

@copilot also allow for vector of tuples or tuples of tuples

[Copilot]

Added support for Vector{Tuple{Symbol,Vararg{Symbol}}} and Tuple{Tuple{Symbol,Vararg{Symbol}},Vararg{Tuple{Symbol,Vararg{Symbol}}}} (and their String equivalents). Updated Symbol_input and String_input type definitions, and modified is_grouped_covariance_input, parse_covariance_groups, and parse_variables_input_to_index to handle these formats. Commit: a13fd2d

[thorek1]

@copilot implement the vectorised approach here and wherever else in the determine_efficient_order functions where you didnt do it yet

[Copilot]

Applied vectorized approach to all remaining locations in determine_efficient_order functions:

1. Second-order covariance section: Replaced nested loops with reshape to nˢ×nˢ matrix and dimensional sums
2. Third-order covariance section: Replaced triple-nested loops with reshape to nˢ×nˢ×nˢ tensor and dimensional sums
3. Propagation loops: Precomputed matrix slices and eliminated copy() calls in both second and third-order covariance sections

All nested loops for dependency detection have been removed. Commit: 291bf89