-
Notifications
You must be signed in to change notification settings - Fork 9
Swap plane Poisson solver with slab Poisson solver in GK IWL simulations & 2x LBO energy conservation #928
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
manauref
wants to merge
43
commits into
main
Choose a base branch
from
poisson_perp_bias
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…on_perp. Unlike the implementation in fem_poisson where one can specify an arbitrary (but aligned with the grid) biased plane, here we specify biased lines, but for the moment it is only restricted to specifyin a line parallel to y (perpendicular to x and z). Not yet tested.
…estricted to a line perpendicular to x and z. Add a unit test, which suggests it seems to work (gkylcas 2c7f34696aa65e552572369d203c031edc2146b4).
… Add GPU unit tests, which pass.
… Add correct logic for parallel smoothing in IWL to the GK field app. Regression tests results are qualitatively and quantiatively similar. A production run will give more confidence.
… a) the 1x and 2x volume kernels were out of date (I didn't change the maxima at all to generate these new kernels), b) When int/corn/surf geo was implemented we used the interior bmag_inv in the GK LBO when we should've used the corner bmag_inv. So here we remove geo_int.bmag_inv (and geo_int.bmag_inv_sq because it's not used anywhere) and add geo_corn.bmag_inv. Now the LBO conserves energy when B(x) and n(x),T(x).
…al magnetic equilibrium.
…sed line in fem_poisson_perp; it wasn't working right when the biased line was placed on the domain boundary. It behaves as expected now, but the new logic isn't totally compatible with TS BCs. That's because the second smoothing changes phi at the biased line (it's no longer 0 if the biased value was 0), so the TS BC of phi will have a non-trivial effect there. Perhaps the thing to do is to apply TS BC to phi before the second smoothing, and used the TS-ed phi as a BC in the smoothing.
…ter the perp solve, apply TS BC to phi on at both boundaries. b) Smooth phi along z with Dirichlet BCs both in the core and the SOL, otherwise the biased limiter value is not preserved (an alternative could be to change fem_parproj so it enforces a biased line value just at the limiter corner in the SOL smoothing).
…use DIRICHLET values from the ghost cell.
…parproj when using Dirichlet BCs.
…values are read from the ghost cell, and adapt the unit test which passes now. Really we should make this an option in the updater because we've gone back and forth on this a couple of times over the last 2 years. For now we simply change this option entirely, for testing (gkylcas 3607635a8b0e4da7d08516021bfe53b72cbd86ab).
…b.com/ammarhakim/gkeyll into poisson_perp_bias-new_IWL_field_solve
…ost cells in the wrong place. We need to fill the ghost cells of the z-global array that goes into fem_parproj. It's currently a bit hacky here, but the simulation appears well behaved. We are presently only applying TS at the lower boundary of the core; if we apply TS on both sides, the solution develops weird gradients in z and the sim crashes in 2-3 micros
…t, and I want to check that it gives similar results (I still don't understand why applying TS at both boundaries behaves so badly).
…ranch with: 1) flattening of the solution in the core ghost before applying TS (not yet tested), 2) some memory fixes in fem_poisson_perp biasing. But I'm seeing cases where biasing works at the bottom but not at the top, and want to switch to laptop for more rapid testing
…n the lower one. Going back to cluster.
…eld to previous state (without the flattening of the ghost before TS, we'll test that after).
…r the ghost cell (since we've gone back and forth between the two, and likely different operations will need one or the other in the future). Adapt unit test so it tests both options. Unit tests pass on CPU, not yet tested on GPU.
…b.com/ammarhakim/gkylzero into poisson_perp_bias-new_IWL_field_solve
…pass on CPU and GPU. Unit test is compute-sanitizer clean.
…using error. Need valgrind from cluster.
…per core boundary. Now rt_gk_d3d_iwl_3x2v_p1 is valgrind clean.
…emory errors, but cuda-gdb found this one.
…tion that creates all the IWL updaters.
…c app, where all core/sol ranges are created. Rename skin/ghost ranges in gyrokinetic and species apps to include local_ so they are consistent with their global counterpart. Checked the following regression tests and they all had unchanged results: rt_gk_mirror_boltz_elc_1x2v_p1 rt_gk_multib_asdex_2x2v_p1 rt_gk_multib_step_2x2v_p1 rt_gk_neut_recycle_1x3v_p1 rt_gk_sheath_1x2v_p1 rt_gk_sheath_2x2v_p1 rt_gk_sheath_3x2v_p1 rt_gk_sheath_fluid_neut_1x2v_p1 rt_gk_tcv_iwl_adapt_source_2x2v_p1 rt_gk_tcv_iwl_adapt_source_3x2v_p1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This goes with PR 93 of gkylcas
Problem
Currently the IWL simulations are using the Poisson solver on planes (
fem_poisson_deflated). This has been shown to break energy conservation. One reason we have continued to do that is that:a) We didn't have the ability to bias the limiter corner in the slab Poisson solver (
fem_poisson_perp).b) We didn't have a working workflow that used the slab Poisson solver and twist-shift BCs (TS BCs).
Here we address both of these points, and deprecate the use of
fem_poisson_deflatedin favor offem_poisson_perp.Solution
Biasing
We implemented the ability to bias the limiter corner in
fem_poisson_perp. Some info is in the DR #898 .Unlike biasing in
fem_poisson, where can bias whole planes in the solver, infem_poisson_perpwe bias lines. That's at the moment we just need to bias the lines at (x,z)=(x_LCFS,z_min) and (x,z)=(x_LCFS,z_max).Tests with biasing were added to the fem_poisson_perp unit test.
Workflow with TS BCs
We implemented a new set of steps to make sure the potential is twist-shift periodic with this new slab solver. These are:
fem_parproj).fem_poisson_perp).Commentary on alternatives
Additional perks
We noticed that the LBO collision operator was not conserving energy in 2x. We fixed that in this branch.
This fix is not the final one, though. Through more testing I found scenarios (large gradients + normNu) in which the LBO still doesn't conserve energy. I think we need to compute nu-weighted moments to truly conserve energy, which we may pursue in another branch.
Tests
This branch is valgrind and compute-sanitizer clean.
Regression tests
Regression tests run, but they are expected to yield slightly different solutions since we are using a different algorithm now.
rt_gk_d3d_iwl_2x2v_p1
rt_gk_tcv_iwl_adapt_source_2x2v_p1
rt_gk_d3d_iwl_3x2v_p1
rt_gk_tcv_iwl_adapt_source_3x2v_p1
Production simulation
We ran a TCV case provided by @Antoinehoff in both main and this branch. Here are some snapshots.