Implementation of feature 720 asym_line

[leovsch]

Implement feature: # 720

Changes proposed in this PR include:

Implementation of feature 720.
Add a new component to support asym_line specification with R, X and C matrix values

Could you please pay extra attention to the points below when reviewing the PR:

I'm a new contributor so pay attention to the coding standards applicable to this project along with achitectural rules etc.

Check list

• Check if the available input/update/output data suffices (check power-grid-model/power_grid_model_c/power_grid_model/include/power_grid_model/auxiliary/ input.hpp / update.hpp / output.hpp or in the documentation directly)
• If not, add the new data format to code_generation/data/attribute_classes/ input.json / update.json / output.json + run code_generation/code_gen.py
• Create a new component in a new power-grid-model/power_grid_model_c/power_grid_model/include/power_grid_model/component/[component].hpp file that at least inherits from Base, but in this case GenericBranch should inherit from Branch
• If necessary: add new enums or exceptions
• Create the necessary unit tests in power-grid-model/tests/cpp_unit_tests/test_[component].cpp
• Add the test_[component].cpp to power-grid-model/tests/cpp_unit_tests/CMakeLists.txt
• Add component to power_grid_model_c/power_grid_model/include/power_grid_model/all_components.hpp
• Not necessary for this component (If necessary update main_core/topology.hpp / input.hpp / output.hpp / update.hpp)
• Add component to code_generation/data/dataset_class_maps/dataset_definitions.json + re-run code_generation/code_gen.py
• Add validation test cases to tests/data
• Update input/update data validator for the new component: src/power_grid_model/validation/validation.py + add corresponding tests

[TonyXiang8787]

Hi @leovsch, thanks for the draft PR. I have put the check list in the description of the PR to track where we are.

[TonyXiang8787]

Hi @leovsch,

Herewith I come back with some suggestions on the implementation of AsymLine class.

Class member

I suggest having the following two class members for the parameters. The y_series_ and y_shunt_ are both 3*3 tensor with complex values.

double i_n_;
double base_i_;
ComplexTensor<asymmetric_t> y_series_;
ComplexTensor<asymmetric_t> y_shunt_;

Construction

The construction of i_n_ and base_i_ is trivial.

For the construction of y_series_, we can do it in the construction body:

if (is_nan(r_na)) {
// if r_na is nan, it means user already does the kron reduction
// we just assign the 3*3 z_series by fill-in with (r + x * imaginary_unit) for each entry
// also fill-in the upper triangle based on symmetry.
}
else {
// if r_na is not nan, the user specifies the neutral conductor parameters, we need to do kron reduction
// in a way that
// z_kl_reduced = z_kl - z_nk * z_nl / z_nn
// k, l = a, b, c
// note: z_nk = z_kn, etc.
// fill-in all the values of 3*3 z_series, also the symmetry
}
y_series_ = inverse(z_series);

We do the similar for y_shunt_. To translate c to y_shunt, use the formula y_shunt = 2 * pi * frequency * c * imaginary_unit

Calculation parameters

Symmetric parameter

To implement sym_calc_param(), we need to calculate y1_series and y1_shunt. Below I give the formula to calculate y1_series. It is the same for y1_shunt.

y_s = average of diagonal of y_series_;
y_m = average of off-diagonal of y_series_;
y1_series = y_s - y_m

You put the y1_series and y1_shunt into calc_param_y_sym.

Asymmetric parameter

To implement asym_calc_param(), there are no helper functions in the Branch to calculate asymmetric parameter in this case. We need to do it to translate y_series_ and y_shunt_ into yff, yft, ytf, ytt depending on the branch connection status.

Please learn the function calc_param_y_sym in Branch and implement something similar. Note all the values are now 3*3 tensor so the division becomes inverse of the tensor. And there is no tap_ratio so the equation is a bit simpler.

power-grid-model/power_grid_model_c/power_grid_model/include/power_grid_model/component/branch.hpp

Lines 205 to 235 in 677c078

	calc_param_y_sym(DoubleComplex const& y_series, // y_series must be converted to the "to" side of the branch
	DoubleComplex const& y_shunt, // y_shunt must be converted to the "to" side of the branch
	DoubleComplex const& tap_ratio) const {
	double const tap = cabs(tap_ratio);
	BranchCalcParam<symmetric_t> param{};
	// not both connected
	if (!branch_status()) {
	// single connected
	if (from_status() || to_status()) {
	DoubleComplex branch_shunt;
	// shunt value
	if (cabs(y_shunt) < numerical_tolerance) {
	branch_shunt = 0.0;
	} else {
	// branch_shunt = y_shunt/2 + 1/(1/y_series + 2/y_shunt)
	branch_shunt = 0.5 * y_shunt + 1.0 / (1.0 / y_series + 2.0 / y_shunt);
	}
	// from or to connected
	param.yff() = from_status() ? (1.0 / tap / tap) * branch_shunt : 0.0;
	param.ytt() = to_status() ? branch_shunt : 0.0;
	}
	}
	// both connected
	else {
	param.ytt() = y_series + 0.5 * y_shunt;
	param.yff() = (1.0 / tap / tap) * param.ytt();
	param.yft() = (-1.0 / conj(tap_ratio)) * y_series;
	param.ytf() = (-1.0 / tap_ratio) * y_series;
	}
	return param;
	}

[leovsch]

First implementation version is here some general remarks and questions:

1. During the implementation I required a 4*4 matrix I've added a new class which makes use of Eigen named ComplexTensor4 to implement this. I can imagine that you would want this somewhere else or with a different name, could anyone give me some insights here?
2. I have added some functions inside the AsymLine class that could serve a more general purpose these being: kron_reduction, average_of_diagonal_of_matrix and average_of_off_diagonal_of_matrix. Is it an idea to move these to a more general place if that is already there?
3. The input parameters I've now implemented would suit my usecase but it would still not fully be equivalent to how opendss handles it. Namely, that you can specify a matrix or two values for the r, x and c values. See their documentation for more information. In my view it would be intuitive for the user to support the OpenDSS way. Can anyone share their thoughts on this?
4. For now I've mainly copied the tests from line and adjusted them to the asym_line implementation. However I have the feeling that I'm missing some context to accuratly write good tests. Could anyone help me out here?

I like working on the project, hope my questions are clear otherwise reach out 👍

[mgovers]

Hi @leovsch ,

I will most likely spend some time tomorrow to review this, but at first glance, you've made the right decisions and followed the correct path. Without digging more into it, here's some initial response on your questions.

First implementation version is here some general remarks and questions:

1. During the implementation I required a 4*4 matrix I've added a new class which makes use of `Eigen` named `ComplexTensor4` to implement this. I can imagine that you would want this somewhere else or with a different name, could anyone give me some insights here?

we try to follow the paradigm "If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck", especially on the things in <power_grid_model/common>. If it has all the properties of a 4D complex tensor, then it's a tensor and ComplexTensor4 is a good name. If it has properties that are not tensor-like (e.g. different transformation behavior), then ComplexMatrix4x4 is probably better. Don't be too scared, though, it's not part of the public interface, so we may change it at any later time anyways.

2. I have added some functions inside the `AsymLine` class that could serve a more general purpose these being: `kron_reduction`, `average_of_diagonal_of_matrix` and `average_of_off_diagonal_of_matrix`. Is it an idea to move these to a more general place if that is already there?

we do have something similar for transformers. That functionality resides in transformer_utils.hpp. You can make a similar branch_utils.hpp or line_utils.hpp for kron_reduction. for the other two, maybe you can come up with a nice name in the common/ subdirectory. Please do add tests for things that are in those locations, though, as they might (and probably will) be reused.

3. The input parameters I've now implemented would suit my usecase but it would still not fully be equivalent to how opendss handles it. Namely, that you can specify a matrix or two values for the r, x and c values. See [their documentation](https://opendss.epri.com/LineCode1.html) for more information. In my view it would be intuitive for the user to support the OpenDSS way. Can anyone share their thoughts on this?

I put it as a topic on the agenda for tomorrow with the rest of the team.

4. For now I've mainly copied the tests from `line` and adjusted them to the `asym_line` implementation. However I have the feeling that I'm missing some context to accuratly write good tests. Could anyone help me out here?

If you like, we can have a call about this. We do have some guidelines, but also a lot of heuristics, so it might be easier to talk to you directly. You can send an email to me (martijn.govers@alliander.com) if that is alright with you, and we can schedule a meeting.

I like working on the project, hope my questions are clear otherwise reach out 👍

That's great to hear! It's very nice to see external contributors working on the project, and we're happy to help you out at any time. Any feedback you have is also highly valued.

Again, thank you for your input so far!

Martijn

[nitbharambe]

1. During the implementation I required a 4*4 matrix I've added a new class which makes use of Eigen named ComplexTensor4 to implement this. I can imagine that you would want this somewhere else or with a different name, could anyone give me some insights here?

1. Agreed with your thinking as the physical interpretation of ComplexTensor4 strays from ComplexTensor. However its generic enough for three_phase_tensor. Adding to @mgovers' comment, maybe if a better name is thought of, you can make an alias in this file's namespace.

Thanks again for the contribution!

[mgovers]

like I said: you're definitely on the right track. a couple ideas.

[mgovers]

formatting can be automatically solved using pre-commit (see https://power-grid-model.readthedocs.io/en/stable/contribution/CONTRIBUTING.html#pre-commit-hooks ; you can apply the format changes retroactively using pre-commit run --all-files and run it before committing using pre-commit install)

Suggested change

	}
	else {
	} else {

[leovsch]

I will try this once I've completely finished the implementation

[mgovers]

code convention: newline at end of file (EOF)

Suggested change

	}
	}

[leovsch]

guess this will be fixed if I use the pre-commit?

[mgovers]

yes indeed, but you might need to run pre-commit run --all-files just to be sure, as it will by default only run on checked-in files during the commit phase.

[mgovers]

newline at EOF

Suggested change

	} // namespace power_grid_model
	} // namespace power_grid_model

[TonyXiang8787]

The input parameters I've now implemented would suit my usecase but it would still not fully be equivalent to how opendss handles it. Namely, that you can specify a matrix or two values for the r, x and c values. See their documentation for more information. In my view it would be intuitive for the user to support the OpenDSS way. Can anyone share their thoughts on this?

@leovsch I have compared with OpenDSS and I could not find the different you mentioned. Maybe can you elaborate a bit more about what exactly are we still missing? Then we can make a choice.

[leovsch]

The input parameters I've now implemented would suit my usecase but it would still not fully be equivalent to how opendss handles it. Namely, that you can specify a matrix or two values for the r, x and c values. See their documentation for more information. In my view it would be intuitive for the user to support the OpenDSS way. Can anyone share their thoughts on this?

@leovsch I have compared with OpenDSS and I could not find the different you mentioned. Maybe can you elaborate a bit more about what exactly are we still missing? Then we can make a choice.

Hi Tony,

In OpenDSS you can per parameter (R, X, C) decide if you want to supply the full matrix or just the 0th and 1st order values.
For example, valid input for OpenDSS might be:

Example 1:
R = full r matrix
X = full x matrix
C = C0, C1

Example 2:
R = R0, R1
X = full x matrix
C = C0, C1

And all posible other permutations of the input format. For now I've implemented only the possibility that would support example 1. I can imagine that you would want to generalize such functionality to all possible permutations of the input format just as in OpenDSS.

Hope this clarifies it a bit for you.

[TonyXiang8787]

The input parameters I've now implemented would suit my usecase but it would still not fully be equivalent to how opendss handles it. Namely, that you can specify a matrix or two values for the r, x and c values. See their documentation for more information. In my view it would be intuitive for the user to support the OpenDSS way. Can anyone share their thoughts on this?

@leovsch I have compared with OpenDSS and I could not find the different you mentioned. Maybe can you elaborate a bit more about what exactly are we still missing? Then we can make a choice.

Hi Tony,

In OpenDSS you can per parameter (R, X, C) decide if you want to supply the full matrix or just the 0th and 1st order values. For example, valid input for OpenDSS might be:

Example 1: R = full r matrix X = full x matrix C = C0, C1

Example 2: R = R0, R1 X = full x matrix C = C0, C1

And all posible other permutations of the input format. For now I've implemented only the possibility that would support example 1. I can imagine that you would want to generalize such functionality to all possible permutations of the input format just as in OpenDSS.

Hope this clarifies it a bit for you.

@leovsch This is a valid point. Thanks for clarification. We need to discuss this and come back to you later.

[TonyXiang8787]

The input parameters I've now implemented would suit my usecase but it would still not fully be equivalent to how opendss handles it. Namely, that you can specify a matrix or two values for the r, x and c values. See their documentation for more information. In my view it would be intuitive for the user to support the OpenDSS way. Can anyone share their thoughts on this?

@leovsch I have compared with OpenDSS and I could not find the different you mentioned. Maybe can you elaborate a bit more about what exactly are we still missing? Then we can make a choice.

Hi Tony,

In OpenDSS you can per parameter (R, X, C) decide if you want to supply the full matrix or just the 0th and 1st order values. For example, valid input for OpenDSS might be:

Example 1: R = full r matrix X = full x matrix C = C0, C1

Example 2: R = R0, R1 X = full x matrix C = C0, C1

And all posible other permutations of the input format. For now I've implemented only the possibility that would support example 1. I can imagine that you would want to generalize such functionality to all possible permutations of the input format just as in OpenDSS.

Hope this clarifies it a bit for you.

Hi @leovsch,

Comeback to your point, we have thoroughly considered different options and decide to go with the attributes you have already defined in this PR. Concretely means:

R = always full r matrix (no r1, r0 possible)
X = always full x matrix (no x1, x0 possible)
C = full c matrix OR C0/C1. If C1 is specified, we use C0/C1 to populate the C matrix and ignore the input c matrix attributes.

From practical applications, if the user want to specify a r1/r0, they will almost certain specify x1/x0 instead of full x matrix, verse versa is also true. In that case, the user can just use normal line instead of asym_line.

We allow C to be specified in both c0/c1 or full c matrix way because this could be a use-case for the user.

[leovsch]

The input parameters I've now implemented would suit my usecase but it would still not fully be equivalent to how opendss handles it. Namely, that you can specify a matrix or two values for the r, x and c values. See their documentation for more information. In my view it would be intuitive for the user to support the OpenDSS way. Can anyone share their thoughts on this?

@leovsch I have compared with OpenDSS and I could not find the different you mentioned. Maybe can you elaborate a bit more about what exactly are we still missing? Then we can make a choice.

Hi Tony,
In OpenDSS you can per parameter (R, X, C) decide if you want to supply the full matrix or just the 0th and 1st order values. For example, valid input for OpenDSS might be:
Example 1: R = full r matrix X = full x matrix C = C0, C1
Example 2: R = R0, R1 X = full x matrix C = C0, C1
And all posible other permutations of the input format. For now I've implemented only the possibility that would support example 1. I can imagine that you would want to generalize such functionality to all possible permutations of the input format just as in OpenDSS.
Hope this clarifies it a bit for you.

Hi @leovsch,

Comeback to your point, we have thoroughly considered different options and decide to go with the attributes you have already defined in this PR. Concretely means:

R = always full r matrix (no r1, r0 possible) X = always full x matrix (no x1, x0 possible) C = full c matrix OR C0/C1. If C1 is specified, we use C0/C1 to populate the C matrix and ignore the input c matrix attributes.

From practical applications, if the user want to specify a r1/r0, they will almost certain specify x1/x0 instead of full x matrix, verse versa is also true. In that case, the user can just use normal line instead of asym_line.

We allow C to be specified in both c0/c1 or full c matrix way because this could be a use-case for the user.

Thanks for the clarifification. I will leave it as is.

[mgovers]

Hi @leovsch , I see that this branch hasn't been updated with main for quite a long time. I recommend doing that. If you have any trouble resolving merge conflicts, feel free to contact me.

[leovsch]

Hi @leovsch , I see that this branch hasn't been updated with main for quite a long time. I recommend doing that. If you have any trouble resolving merge conflicts, feel free to contact me.

I did a rebase of my fork last week monday, I did not have any merge conflicts, does that typically not show up here? Or do you mean that since last monday a lot has changed?

[mgovers]

Hi @leovsch , I see that this branch hasn't been updated with main for quite a long time. I recommend doing that. If you have any trouble resolving merge conflicts, feel free to contact me.

I did a rebase of my fork last week monday, I did not have any merge conflicts, does that typically not show up here? Or do you mean that since last monday a lot has changed?

ohhh that's great. I did not realize that you only did it last monday. Great that you did not have merge conflicts. Usually, the amount of merge conflicts is small for things like a new component, but you never know 😬

[leovsch]

https://github.com/PowerGridModel/power-grid-model/blob/f4927bf94b6b32790400f596f8103195aa6689fe/src/power_grid_model/validation/rules.py#L757C1-L757C18

While using this function for the new asym_line input validation function I ran into a potential issue (or my misterpretation of this function).

Consider the following input data for aym_line:

asym_line = initialize_array(DatasetType.input, ComponentType.asym_line, 2)
asym_line["id"] = [55, 56]
asym_line["from_node"] = [0, 1]
asym_line["to_node"] = [1, 2]
asym_line["from_status"] = [1, 1]
asym_line["to_status"] = [1, 1]
asym_line["r_aa"] = [-1, 2]
asym_line["r_ba"] = [-1, 2]
asym_line["r_bb"] = [-1, 2]
asym_line["r_ca"] = [-1, 2]
asym_line["r_cb"] = [-1, 2]
asym_line["r_cc"] = [-1, 2]
asym_line["r_na"] = [-1, 2]
asym_line["r_nb"] = [-1, 2]
asym_line["r_nc"] = [-1, 2]
asym_line["r_nn"] = [-1, 2]
asym_line["x_aa"] = [-1, 2]
asym_line["x_ba"] = [-1, 2]
asym_line["x_bb"] = [-1, 2]
asym_line["x_ca"] = [-1, 2]
asym_line["x_cb"] = [-1, 2]
asym_line["x_cc"] = [-1, 2]
asym_line["x_na"] = [-1, 2]
asym_line["x_nb"] = [-1, 2]
asym_line["x_nc"] = [-1, 2]
asym_line["x_nn"] = [-1, 2]
asym_line["c_aa"] = [-1, np.nan]
asym_line["c_ba"] = [-1, np.nan]
asym_line["c_bb"] = [-1, np.nan]
asym_line["c_ca"] = [-1, np.nan]
asym_line["c_cb"] = [-1, np.nan]
asym_line["c_cc"] = [-1, np.nan]
asym_line["c_na"] = [-1, np.nan]
asym_line["c_nb"] = [-1, np.nan]
asym_line["c_nc"] = [-1, np.nan]
asym_line["c_nn"] = [-1, np.nan]
asym_line["c0"]   = [-1, np.nan]
asym_line["c1"]   = [-1, np.nan]
asym_line["i_n"] =  [50, 50]

by calling the none_missing function as follows:

none_missing(data, ComponentType.asym_line, ["c_aa", "c_ba"], 1)

I would expect the function to return two `MissingValueError' instances that look like this:

MissingValueError(component=ComponentType.asym_line, field="c_aa", ids=[56])
MissingValueError(component=ComponentType.asym_line, field="c_ba", ids=[56])

Because this is the component at index 1 with id 56 and has missing values for the fields c_aa and c_ba.
However when I use this function in my validation function I get the following output:

MissingValueError(component=ComponentType.asym_line, field="c_aa", ids=[55, 56])
MissingValueError(component=ComponentType.asym_line, field="c_ba", ids=[55, 56])

This is unexpected as id 55 does not have a missing value for c_aa and c_ba.
Is this behaviour as expected? Or is this a potential bug? Can I use this function as described or should I implement something new?

[mgovers]

This is unexpected as id 55 does not have a missing value for c_aa and c_ba.
Is this behaviour as expected? Or is this a potential bug? Can I use this function as described or should I implement something new?

Hi @leovsch , I can confirm that this is a bug in the none_missing function. I will fix and test it myself.

EDIT: I created #902 to this end.

[mgovers]

This is unexpected as id 55 does not have a missing value for c_aa and c_ba.
Is this behaviour as expected? Or is this a potential bug? Can I use this function as described or should I implement something new?

Hi @leovsch , I can confirm that this is a bug in the none_missing function. I will fix and test it myself.

EDIT: I created #902 to this end.

In the meantime, I recommend you add the check for required asym_line values to validate_required_values instead of where they are now ( https://github.com/PowerGridModel/power-grid-model/blob/main/src/power_grid_model/validation/validation.py#L286-L410 )

[leovsch]

This is unexpected as id 55 does not have a missing value for c_aa and c_ba.
Is this behaviour as expected? Or is this a potential bug? Can I use this function as described or should I implement something new?

Hi @leovsch , I can confirm that this is a bug in the none_missing function. I will fix and test it myself.
EDIT: I created #902 to this end.

In the meantime, I recommend you add the check for required asym_line values to validate_required_values instead of where they are now ( https://github.com/PowerGridModel/power-grid-model/blob/main/src/power_grid_model/validation/validation.py#L286-L410 )

Hi @mgovers ,

Thanks for creating the issue. I've looked into your suggestion and I think the values that are always required are already in there. For the other ones I think this is kind of a special case. As some values might be optional depending on other values.

For example,

When a user supplied the C1 and C0, all the values for the full c matrix are not required anymore but either one should be supllied i.e. C1 and C0 or the full C_matrix.
Furthermore, the neutral values for the r and x matrix are also optional but they should either all be there or none should be there. So how I've implemented it now is that when a user supplies one of the neutral values for the r and x matrices the other ones become required. e.g. when the user supplies values for r_na and r_nb but no values for r_nc and r_nn this would be invalid. So the intention is that when the user supplies one of the optional values all other optional values should be supplied as well.

Looking at the validate_required_values function this seems to mainly focus on the values that are always required, hence, I did not see this as a proper fit.

What are your thoughts on this?

[mgovers]

This is unexpected as id 55 does not have a missing value for c_aa and c_ba.
Is this behaviour as expected? Or is this a potential bug? Can I use this function as described or should I implement something new?

Hi @leovsch , I can confirm that this is a bug in the none_missing function. I will fix and test it myself.
EDIT: I created #902 to this end.

In the meantime, I recommend you add the check for required asym_line values to validate_required_values instead of where they are now ( https://github.com/PowerGridModel/power-grid-model/blob/main/src/power_grid_model/validation/validation.py#L286-L410 )

Hi @mgovers ,

Thanks for creating the issue. I've looked into your suggestion and I think the values that are always required are already in there. For the other ones I think this is kind of a special case. As some values might be optional depending on other values.

For example,

When a user supplied the C1 and C0, all the values for the full c matrix are not required anymore but either one should be supllied i.e. C1 and C0 or the full C_matrix. Furthermore, the neutral values for the r and x matrix are also optional but they should either all be there or none should be there. So how I've implemented it now is that when a user supplies one of the neutral values for the r and x matrices the other ones become required. e.g. when the user supplies values for r_na and r_nb but no values for r_nc and r_nn this would be invalid. So the intention is that when the user supplies one of the optional values all other optional values should be supplied as well.

Looking at the validate_required_values function this seems to mainly focus on the values that are always required, hence, I did not see this as a proper fit.

What are your thoughts on this?

Hi @leovsch ,

I see what you mean. We do have something similar for power sensors: either power_sigma and/or both (p_sigma and q_sigma) should be provided. See also:

• the matrix in https://power-grid-model.readthedocs.io/en/stable/user_manual/components.html#id31
• and _process_power_sigma_and_p_q_sigma in https://github.com/PowerGridModel/power-grid-model/blob/main/src/power_grid_model/validation/validation.py#L247C5-L283 .

However, I have found a bug in the latter (closely related to the bug in none_missing bug you found), so I would say, hold off on implementing it for now and wait until the fix for both _process_power_sigma_and_p_q_sigma and none_missing is in place.

I have already found and implemented a sustainable solution including tests for the none_missing. I am now looking at the one in _process_power_sigma_and_p_q_sigma before I create a PR for that.