GhostingFunctor example for adding ghost elements to DistributedMesh

[ajamar1]

Hello,

Does anyone in the community have a simple example of using a GhostingFunctor to add ghost elements for a DistributedMesh? I can't seem to get it working. I essentially want to add all boundary elements as ghost elements on every processor. Miscellaneous example 9 (https://github.com/libMesh/libmesh/tree/master/examples/miscellaneous/miscellaneous_ex9) uses a GhostingFunctor, but it attaches to the DofMap as a coupling functor. I want to utilize DistributedMesh::add_ghosting_functor(), which I think is the preferred approach over DistributedMesh::add_extra_ghost_elem().

As an alternative, I could comment up a simple example that I think is not working for me, if that would be useful.

Thanks to anyone who can point me in the right direction.

Adam

[roystgnr]

By ghost elements, you mean you merely want the boundary elements to be geometrically present, and you don't need to retain any algebraic information on them? The mesh API is the correct one for that, but if you really do need algebraic information too then DofMap::add_algebraic_ghosting_functor() is what you need to use.

But the most likely mistake to make (one which required changes in even our example codes) is hinted at by an imprecision in your language: even with a GhostingFunctor, ghost elements can never be added. They can only be prevented from being removed when the mesh goes distributed, or transferred along with the local elements they're tied to when the mesh gets redistributed. In our examples where we originally would attach periodic boundary conditions to a system after it's mesh had already been prepared for use, for example, it was too late - the periodic pairs of elements had already been decoupled and the remote ones deleted on every processor and so by the time the PBCs were added they couldn't be recovered. In your case, if you were to write a ghosting functor that loops over all elements looking for boundary elements to retain, but you only attached it after the mesh has been distributed, it would never see any of the already-deleted boundary elements and so would never be able to report that they shouldn't have been deleted. You'd have to either add the ghosting functor before the mesh is ever prepared, or reserialize the mesh temporarily after adding it but before relying on it.

If neither of those are the source of the error ... nothing else comes to mind immediately. I'd love to see an example in that case.

[ajamar1]

Roy,

I was looking back at my post and realized that I didn't reply to your comment directly, but instead wrote my own "answer." Sorry about that. My second post was meant to be a reply to you, and it includes an example code that is not working for me.

Thanks and sorry for the confusion.

Adam

[ajamar1]

Roy,

Thanks, as always, for the thorough response. I do think I am missing something fundamental, or my C++ inexperience is getting the best of me. First to your question....

By ghost elements, you mean you merely want the boundary elements to be geometrically present, and you don't need to retain any algebraic information on them?

Correct. I only need the geometric information. I am attaching the GhostingFunctor before I read the mesh from file, which I think is early enough. I have attached an example (using libMesh 1.6.2) where I read a mesh three times.

1. Without a GhostingFunctor
2. With a GhostingFunctor that uses libMesh's default inputs for element range in the GhostingFunctor
3. With a GhostingFunctor that I force to loop over all elements in the mesh

For each, I write out the number of ghost elements on each processor, and all cases are identical. I have attached the code (file extension changed to allow upload), and the output on four processors is:

Case 0: No GhostingFunctor

Processor, # ghost elements
0 1045
1 735
2 1123
3 617

Case 1: MyGhostingFunctor_1

Processor, # added ghost elements
0 0
4294967295 0
1 0
4294967295 0
2 0
4294967295 0
3 0
4294967295 0

Processor, # ghost elements
0 1045
1 735
2 1123
3 617

Case 2: MyGhostingFunctor_2

Processor, # added ghost elements
0 1871
4294967295 2650
1 1947
4294967295 2650
2 2081
4294967295 2650
3 2051
4294967295 2650

Processor, # ghost elements
0 1045
1 735
2 1123
3 617

Perhaps my check for the number of ghost elements is erroneous?

std::distance(mesh.ghost_elements_begin(),mesh.ghost_elements_end())

Also notice that the GhostingFunctor operator() gets called twice per processor during the read: once with the proper processor id and the second time with processor id = 4294967295. This makes me think I have some bad C++.

I apologize if this is user incompetence.

Thanks,

Adam

foo.cc.txt