Add 'Parallel SR-IOV configuration' design document #479

e0ne · 2023-07-19T06:13:59Z

No description provided.

github-actions · 2023-07-19T06:14:11Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

coveralls · 2023-07-19T06:21:24Z

Pull Request Test Coverage Report for Build 7611451473

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 24.394%

Totals
Change from base Build 7570704762:	0.0%
Covered Lines:	2426
Relevant Lines:	9945

💛 - Coveralls

README.md

doc/design/parallel-node-config.md

adrianchiris · 2023-07-20T13:35:27Z

doc/design/parallel-node-config.md

+
+### API Extensions
+
+#### Option 1: extend existing CR SriovNetworkPoolConfig


+1 on this option from me.

@e0ne if you are OK with it, lets remove "Option 1: " string here

and place option2 and 3 under a new subsection: "Alternative APIs"

unless you feel that we should go with a different approach in this design doc.

also others feedback is appreciated on which approach is preferable.
i think @SchSeba also prefers this option.

github-actions · 2023-07-24T09:10:19Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

SchSeba

nice to see this work!
I am really expecting for this feature

SchSeba · 2023-07-27T09:47:36Z

doc/design/parallel-node-config.md

+AnnoDraining        = "Draining_Complete"
+```
+
+Drain controller will watch for node annotation changes, `SriovNetworkPoolConfig` and `SriovNetworkNodeState` changes:


+1
I think we should stop using the node this way we can also reduce the RBAC for the config-daemon to only watch for sriovNetworkNodeState

doc/design/parallel-node-config.md

SchSeba · 2023-07-27T09:52:44Z

doc/design/parallel-node-config.md

+    // OvsHardwareOffloadConfig describes the OVS HWOL configuration for selected Nodes
+    OvsHardwareOffloadConfig OvsHardwareOffloadConfig `json:"ovsHardwareOffloadConfig,omitempty"`
+    // NodeSelectorTerms is a list of node selectors to apply SriovNetworkPoolConfig
+    NodeSelectorTerms *v1.NodeSelector `json:"nodeSelectorTerms,omitempty"`


maybe just

nodeSelector: matchLabels: node-role.kubernetes.io/worker: ""

without the Terms

I think the idea was to re-use this:

https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#nodeselector-v1-core

doc/design/parallel-node-config.md

SchSeba · 2023-07-27T10:04:09Z

doc/design/parallel-node-config.md

+* introduce new API changes to support pool of nodes to be drained in a parallel:
+  at this phase we introduce new API from one of the proposed options above and modify Drain controller to watch for
+  the specified CRs and proceed drain in a parallel per node pool configuration
+* drop `NodeDrainAnnotation` usage and move this annotation values into the `SriovNetworkNodeState` object


agree we should move this as part of the first change.

we much check how this will work in case the operator upgrade is done in the middle of the configuration.

SchSeba · 2023-07-27T10:12:34Z

doc/design/parallel-node-config.md

+All phases should be implemented one-by-one in a separate PRs in the order above.
+
+### Upgrade & Downgrade considerations
+This feature introduces changes to CRDs which means CRDs should be applied during upgrade or downgrade 


I think we should have a support for this as we have users that runs the sriov operator.
we need to be sure a case where the operator gets updated in the middle of a drain is something we support and are able to recover from it

SchSeba · 2023-07-27T13:09:32Z

One more comment is we need to see how the machine config will work with parallel node draining for openshift

api/v1/sriovnetworkpoolconfig_types.go

github-actions · 2023-08-01T20:14:04Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

doc/design/parallel-node-config.md

adrianchiris · 2023-08-21T11:33:10Z

doc/design/parallel-node-config.md

+
+## Proposal
+
+Introduce nodes pool drain configuration to meet following requirements:


i really prefer not to copy the goals into proposal.

maybe something like:

Introduce nodes pool drain configuration and controller to meet Goals targets.

doc/design/parallel-node-config.md

adrianchiris · 2023-08-21T11:51:36Z

doc/design/parallel-node-config.md

+    // OvsHardwareOffloadConfig describes the OVS HWOL configuration for selected Nodes
+    OvsHardwareOffloadConfig OvsHardwareOffloadConfig `json:"ovsHardwareOffloadConfig,omitempty"`
+    // NodeSelectorTerms is a list of node selectors to apply SriovNetworkPoolConfig
+    NodeSelectorTerms *v1.NodeSelector `json:"nodeSelectorTerms,omitempty"`


I think the idea was to re-use this:

https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#nodeselector-v1-core

doc/design/parallel-node-config.md

github-actions · 2023-08-21T12:41:01Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

github-actions · 2023-08-21T12:41:34Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

doc/design/parallel-node-config.md

github-actions · 2023-09-08T11:19:56Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

doc/design/parallel-node-config.md

adrianchiris

@e0ne gave this one another look.

once my remaining comments are addressed, consider it LGTM from my side.

github-actions · 2023-09-12T11:41:26Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

SchSeba

great work I am really waiting for this feature to go in!

I left some small comments

doc/design/parallel-node-config.md

SchSeba · 2023-09-13T13:05:38Z

doc/design/parallel-node-config.md

+// +kubebuilder:rbac:groups="",resources=SriovNetworkPoolConfig,verbs=get;list;watch;update;patch
+// +kubebuilder:rbac:groups="",resources=SriovNetworkNodeState,verbs=get;list;watch;update;patch
+```
+


+1 to have it numerated

doc/design/parallel-node-config.md

SchSeba · 2023-09-13T13:15:47Z

doc/design/parallel-node-config.md

+`SriovNetworkNodeStates` update.
+
+Config daemon will be responsible for setting `SriovNetworkNodeState.DrainStatus=Drain_Required` and
+`SriovNetworkNodeState.DrainStatus=DrainComplete` only. It will simplify its implementation. 


I don't think the DrainComplete will be set by the config-daemon

How controllel will know that drain finished? It's less complicated to to it in a config daemon

SchSeba · 2023-09-13T13:17:22Z

doc/design/parallel-node-config.md

+type SriovNetworkNodeStateStatus struct {
+	Interfaces    InterfaceExts `json:"interfaces,omitempty"`
+    // +kubebuilder:validation:Enum=Idle;Draining,Draining_MCP_Paused,Succeeded,Failed,InProgress
+	DrainStatus    string        `json:"drainStatus,omitempty"`


can we have this as a struct? that is a string but just for easier connection

doc/design/parallel-node-config.md

SchSeba · 2023-09-21T17:29:56Z

doc/design/parallel-node-config.md

+    // OvsHardwareOffloadConfig describes the OVS HWOL configuration for selected Nodes
+    OvsHardwareOffloadConfig OvsHardwareOffloadConfig `json:"ovsHardwareOffloadConfig,omitempty"`
+    // NodeSelectorTerms is a list of node selectors to apply SriovNetworkPoolConfig
+    NodeSelectorTerms *v1.NodeSelectorTerm `json:"nodeSelectorTerms,omitempty"`


can we change this to NodeSelector only like this?

https://docs.openshift.com/container-platform/4.10/rest_api/machine_apis/machineconfigpool-machineconfiguration-openshift-io-v1.html#spec-machineconfigselector

IMO, NodeSelectorTerm is more adjustable

+1 to sticking with standard k8s NodeSelectorTerms.
these can be translated to machineconfig selector if needed.

WDYT?

SchSeba · 2023-09-21T17:31:24Z

doc/design/parallel-node-config.md

+    OvsHardwareOffloadConfig OvsHardwareOffloadConfig `json:"ovsHardwareOffloadConfig,omitempty"`
+    // NodeSelectorTerms is a list of node selectors to apply SriovNetworkPoolConfig
+    NodeSelectorTerms *v1.NodeSelectorTerm `json:"nodeSelectorTerms,omitempty"`
+    DrainConfig DrainConfigSpec `json:"drainConfig,omitempty"`


I don't think we need another struct can we have something like

maxUnavailable integer-or-string maxUnavailable specifies the percentage or constant number of machines that can be updating at any given time. default is 1.

It's introduced in case if we decide add more drain-related config options (e.g. false=true) so we won't need to change API in the future

+1 on having this a stuct for future extensability.

can you please change to something like what we have in the PR?

// SriovNetworkPoolConfigSpec defines the desired state of SriovNetworkPoolConfig type SriovNetworkPoolConfigSpec struct { // OvsHardwareOffloadConfig describes the OVS HWOL configuration for selected Nodes OvsHardwareOffloadConfig OvsHardwareOffloadConfig `json:"ovsHardwareOffloadConfig,omitempty"` // nodeSelector specifies a label selector for Nodes NodeSelector *metav1.LabelSelector `json:"nodeSelector,omitempty"` // maxUnavailable defines either an integer number or percentage // of nodes in the pool that can go Unavailable during an update. // // A value larger than 1 will mean multiple nodes going unavailable during // the update, which may affect your workload stress on the remaining nodes. // Drain will respect Pod Disruption Budgets (PDBs) such as etcd quorum guards, // even if maxUnavailable is greater than one. MaxUnavailable *intstr.IntOrString `json:"maxUnavailable,omitempty"` }

SchSeba · 2023-09-21T18:00:02Z

doc/design/parallel-node-config.md

+Operator. To not introduce breaking changes we have to split this effort to several phases:
+* implement new Drain controller without user-facing API changes:
+  it will proceed only one node configuration at the same time and doesn't require API changes
+* drop `SriovNetworkNodeState` usage and move this annotation values into the `SriovNetworkNodeState` object


I think here is to drop the annotation from the node object

github-actions · 2023-09-28T08:02:51Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

SchSeba · 2023-10-01T16:25:57Z

doc/design/parallel-node-config.md

+      operator: "Exists"
+      drainConfig:
+        maxUnavailable: 5
+        ovsHardwareOffloadConfig: {}


the indentation here is not right I think

Also, drainConfig is already defined for this resource. Right?

yea i believe this needs to be removed.

SchSeba · 2023-10-01T16:26:02Z

doc/design/parallel-node-config.md

+name: default
+namespace: network-operator
+spec:
+priority: 0


the indentation here is not right I think

zeeke · 2023-10-20T14:08:55Z

@e0ne , @SchSeba, I was thinking of the following scenario:

Given a cluster with 5 worker nodes (A, B, C, D, E)
And two SriovNetworkPoolConfigs:
  pool1 that targets A,B,C, priority 1, maxParallelNodeConfiguration 1
  pool2 that targets C,D,E, priority 99, maxParallelNodeConfiguration 2

When the user creates a policy that applies to C,D,E
Then C, D, E starts configuring immediately, because:
- C belongs to pool1 (as it has the highest priority), where there is no other node configuring ATM
- D and E belongs to pool2 and it has maxParalleNodeConfiguration = 2

If my considerations are right, please add this scenario to a "Examples" section.

adrianchiris · 2023-10-22T07:50:03Z

@e0ne , @SchSeba, I was thinking of the following scenario:

Given a cluster with 5 worker nodes (A, B, C, D, E)
And two SriovNetworkPoolConfigs:
  pool1 that targets A,B,C, priority 1, maxParallelNodeConfiguration 1
  pool2 that targets C,D,E, priority 99, maxParallelNodeConfiguration 2

When the user creates a policy that applies to C,D,E
Then C, D, E starts configuring immediately, because:
- C belongs to pool1 (as it has the highest priority), where there is no other node configuring ATM
- D and E belongs to pool2 and it has maxParalleNodeConfiguration = 2

If my considerations are right, please add this scenario to a "Examples" section.

+1 on emphasizing that a node may belong to at most a single pool (the one with the most important priority)

SchSeba · 2023-10-22T11:16:08Z

yep great example!

SchSeba · 2023-10-29T17:44:26Z

also, another point is if the node selector is good for 2 pools and the priority is equal we will select the first pool in alphabetic order

adrianchiris · 2024-01-22T12:25:21Z

doc/design/parallel-node-config.md

+## Motivation
+SR-IOV Network Operator configures SR-IOV one node at a time and one nic at a same time. That means we’ll need to wait
+hours or even days to configure all NICs  on large cluster deployments. With multi-NICs deployments Operator configures
+NICs one by one on each node which  leads to a lot of unnecessary node drain calls during SR-IOV configuration. Also


are we really saving up on drain calls ?

even currently sriov-network-config-daemon will configure all relevant nic in the node. it just does it serially after draining.

adrianchiris · 2024-01-22T12:38:47Z

doc/design/parallel-node-config.md

+
+#### Extend existing CR SriovNetworkPoolConfig
+SriovNetworkPoolConfig is used only for OpenShift to provide configuration for
+OVS Hardware Offloading. We can extend it to add configuration for the drain


can you add some info that SriovNetworkPoolConfig will now be relevant for all cluster types (kubernetes and openshift)

adrianchiris · 2024-01-22T12:41:02Z

doc/design/parallel-node-config.md

+
+### Upgrade & Downgrade considerations
+After operator upgrade we have to support `sriovnetwork.openshift.io/state` node annotation to
+`SriovNetworkNodeState.Status.DrainStatus` migration. This logic will be implemented in a Drain controller and should be


shouldnt we implement the logic in config daemon as it is the one who triggers drain flow ?
just so we dont have race condition where config daemon updates drain status AND controller tries to migrate the annotation

github-actions · 2024-01-22T12:56:16Z

Thanks for your PR,
To run vendors CIs use one of:

/test-all: To run all tests for all vendors.
/test-e2e-all: To run all E2E tests for all vendors.
/test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs use one of:

/skip-all: To skip all tests for all vendors.
/skip-e2e-all: To skip all E2E tests for all vendors.
/skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
Best regards.

SchSeba · 2024-02-01T13:36:14Z

doc/design/parallel-node-config.md

+drain and use `drain lock` in config daemon anymore. The overall drain process will be covered by the following states:
+
+```golang
+DrainIdle        = "Idle"


Hi @e0ne can you please change this to what we have in the PR?

should be

NodeDrainAnnotation = "sriovnetwork.openshift.io/state" NodeStateDrainAnnotation = "sriovnetwork.openshift.io/desired-state" NodeStateDrainAnnotationCurrent = "sriovnetwork.openshift.io/current-state" DrainIdle = "Idle" DrainRequired = "Drain_Required" RebootRequired = "Reboot_Required" Draining = "Draining" DrainComplete = "DrainComplete"

SchSeba · 2024-02-01T13:39:23Z

doc/design/parallel-node-config.md

+    OvsHardwareOffloadConfig OvsHardwareOffloadConfig `json:"ovsHardwareOffloadConfig,omitempty"`
+    // NodeSelectorTerms is a list of node selectors to apply SriovNetworkPoolConfig
+    NodeSelectorTerms *v1.NodeSelectorTerm `json:"nodeSelectorTerms,omitempty"`
+    DrainConfig DrainConfigSpec `json:"drainConfig,omitempty"`


can you please change to something like what we have in the PR?

// SriovNetworkPoolConfigSpec defines the desired state of SriovNetworkPoolConfig type SriovNetworkPoolConfigSpec struct { // OvsHardwareOffloadConfig describes the OVS HWOL configuration for selected Nodes OvsHardwareOffloadConfig OvsHardwareOffloadConfig `json:"ovsHardwareOffloadConfig,omitempty"` // nodeSelector specifies a label selector for Nodes NodeSelector *metav1.LabelSelector `json:"nodeSelector,omitempty"` // maxUnavailable defines either an integer number or percentage // of nodes in the pool that can go Unavailable during an update. // // A value larger than 1 will mean multiple nodes going unavailable during // the update, which may affect your workload stress on the remaining nodes. // Drain will respect Pod Disruption Budgets (PDBs) such as etcd quorum guards, // even if maxUnavailable is greater than one. MaxUnavailable *intstr.IntOrString `json:"maxUnavailable,omitempty"` }

SchSeba · 2024-02-07T15:05:03Z

I spoke with @e0ne I am taking the work on the design doc for the parallel drain so please continue the collaboration here #626

adrianchiris reviewed Jul 20, 2023

View reviewed changes

README.md Outdated Show resolved Hide resolved

adrianchiris requested changes Jul 20, 2023

View reviewed changes

e0ne force-pushed the parallel-nodes-design branch from 03b497e to 8275dd2 Compare July 24, 2023 09:10

SchSeba reviewed Jul 27, 2023

View reviewed changes

adrianchiris reviewed Jul 31, 2023

View reviewed changes

api/v1/sriovnetworkpoolconfig_types.go Outdated Show resolved Hide resolved

e0ne force-pushed the parallel-nodes-design branch from 8275dd2 to afea85f Compare August 1, 2023 20:13

wizhaoredhat reviewed Aug 3, 2023

View reviewed changes

doc/design/parallel-node-config.md Outdated Show resolved Hide resolved

doc/design/parallel-node-config.md Outdated Show resolved Hide resolved

doc/design/parallel-node-config.md Outdated Show resolved Hide resolved

SchSeba reviewed Aug 9, 2023

View reviewed changes

doc/design/parallel-node-config.md Outdated Show resolved Hide resolved

e0ne mentioned this pull request Aug 9, 2023

Drain controller #488

Closed

adrianchiris reviewed Aug 21, 2023

View reviewed changes

e0ne force-pushed the parallel-nodes-design branch from afea85f to 0918908 Compare August 21, 2023 12:40

e0ne force-pushed the parallel-nodes-design branch from 0918908 to 53fcc49 Compare August 21, 2023 12:41

adrianchiris reviewed Sep 5, 2023

View reviewed changes

doc/design/parallel-node-config.md Show resolved Hide resolved

e0ne force-pushed the parallel-nodes-design branch from 53fcc49 to 3c42bfc Compare September 8, 2023 11:19

adrianchiris reviewed Sep 11, 2023

View reviewed changes

doc/design/parallel-node-config.md Outdated Show resolved Hide resolved

adrianchiris reviewed Sep 11, 2023

View reviewed changes

doc/design/parallel-node-config.md Outdated Show resolved Hide resolved

adrianchiris reviewed Sep 11, 2023

View reviewed changes

e0ne force-pushed the parallel-nodes-design branch from 3c42bfc to d3b2c56 Compare September 12, 2023 11:41

e0ne requested a review from SchSeba September 12, 2023 11:41

SchSeba reviewed Sep 21, 2023

View reviewed changes

e0ne force-pushed the parallel-nodes-design branch from d3b2c56 to 7601e18 Compare September 28, 2023 08:02

SchSeba reviewed Oct 1, 2023

View reviewed changes

SchSeba mentioned this pull request Nov 25, 2023

Drain controller #552

Closed

SchSeba mentioned this pull request Dec 5, 2023

Parallel drain #555

Merged

adrianchiris reviewed Jan 22, 2024

View reviewed changes

e0ne force-pushed the parallel-nodes-design branch from 7601e18 to d505c92 Compare January 22, 2024 12:56

Add 'Parallel SR-IOV configuration' design document

d505c92

SchSeba reviewed Feb 1, 2024

View reviewed changes

SchSeba closed this Feb 7, 2024


		### API Extensions

		#### Option 1: extend existing CR SriovNetworkPoolConfig


		## Proposal

		Introduce nodes pool drain configuration to meet following requirements:

Add 'Parallel SR-IOV configuration' design document #479

Add 'Parallel SR-IOV configuration' design document #479

Conversation

e0ne commented Jul 19, 2023

github-actions bot commented Jul 19, 2023

coveralls commented Jul 19, 2023 • edited Loading

Pull Request Test Coverage Report for Build 7611451473

💛 - Coveralls

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 24, 2023

SchSeba left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SchSeba commented Jul 27, 2023

github-actions bot commented Aug 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

github-actions bot commented Sep 8, 2023

adrianchiris left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 12, 2023

SchSeba left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeeke commented Oct 20, 2023 • edited Loading

adrianchiris commented Oct 22, 2023

SchSeba commented Oct 22, 2023

SchSeba commented Oct 29, 2023

adrianchiris Jan 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrianchiris Jan 22, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Jan 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SchSeba commented Feb 7, 2024

coveralls commented Jul 19, 2023 •

edited

Loading

zeeke commented Oct 20, 2023 •

edited

Loading

adrianchiris Jan 22, 2024 •

edited

Loading

adrianchiris Jan 22, 2024 •

edited

Loading