Nemesis Solution User Object

[robtclay]

I saw that recently support was added to solution user objects for nemesis inputs with: #32297

I updated my MOOSE install to try this out and I am noticing a few issues that I'd like to get help with.

The first issue is that it looks like the nemesis files were changed to be fileName.n.#.# rather than fileName.e.#.# and it seems like Paraview doesn't recognize the new format? I tried loading one of these files and it wasn't able to decide on what reader to use. It also required me changing the default file search from supported files to all files since it doesn't seem to support the new file format natively.

Is there a reader that is recommended for the new file format or some sort of addon that is needed for Paraview to have it recognize the files? I could write a script to change all the file names back to their fileName.e.#.# format, but I figure that shouldn't be necessary.

The second issue is related to actually trying to use the functionality. I ran the first step of my problem with a distributed mesh and nemesis output. Then if I try to run the 2nd step, also with distributed mesh, nemesis output, and now a SUO loading in the nemesis files from the previous step, it gives errors.

The errors are all about failing to access the data for my variables and look similar to this one:
Failed to access the data for variable 'eta_f' at point (x,y,z)=(-90340.3, -5572.8, 0) in the 'solution_uo' SolutionUserObjectBase

eta_f is one of my two phase field variables in the system and is set as part of the initial conditions in the first step and imported into my 2nd step with the solution user objects. Also potentially worth noting, we set this using the EBSDReader functionality in the first step. and use the EBSDMeshGenerator for creating the mesh in both steps.

I know that #32411 is open and it seems related, but I am not using any discontinuous galerkin variables.

As part of my testing I found that if my mesh in the second step was not set to distributed, or if I set force_replicated_source_mesh = true in the SUO of my 2nd step, it is able to run successfully with the nemesis output from the first step. It also works fine if I just do an exodus output from the first step and load that in through SUO like I did before this new update.

Please let me know if I am missing anything or if there are other things I should try for either of these, thanks!

[GiudGiud]

Hello

The first issue is that it looks like the nemesis files were changed to be fileName.n.#.# rather than fileName.e.#.# and it seems like Paraview doesn't recognize the new format?

When it asks for the reader in Paraview, please pick the IOSS reader. THe exodus legacy reader might also be able to do it

Is there a reader that is recommended for the new file format or some sort of addon that is needed for Paraview to have it recognize the files? I could write a script to change all the file names back to their fileName.e.#.# format, but I figure that shouldn't be necessary.

I think we might just allow you to output nemesis with .e using a parameter in MOOSE. Seems easier.

The errors are all about failing to access the data for my variables and look similar to this one:
Failed to access the data for variable 'eta_f' at point (x,y,z)=(-90340.3, -5572.8, 0) in the 'solution_uo' SolutionUserObjectBase

Is the partitioning of the two simulations exactly the same? We need the partitions to overlap exactly if the solutionUO is distributed and not serialized.

and use the EBSDMeshGenerator for creating the mesh in both steps.

in the same way in both simulations? Having partitions match for two different meshes is likely challenging

You can use Debug/output_process_domains = true to see the partitionining of the mesh in each simulation

[robtclay]

IOSS Reader does seem to work, so thank you for that information. Nemesis used to output with .e file format so this is only a recent change. I agree some way to output with the .e file format from the beginning would be better than needing a custom filename change script.

The meshes that they're loading are from the same file. I used the debug option like you suggested and it does appear to be doing a different PID distribution for the mesh the 2nd time which is what I expect is the issue. Is there a way to force the distribution to be the same? I am using the same number of cores for both steps.

[GiudGiud]

hello

Is there a way to force the distribution to be the same?

The only way I can think of is to write a custom partitioner.

The easier way would be to re-run both simulations with a partitioner that gives the same result on both meshes. For example a GridPartitioner
https://mooseframework.inl.gov/source/partitioner/GridPartitioner.html

If you don't want to delve into partitioners, you could load the results through a distributed multiapps, then transfer to the parent simulation with the new mesh. The partitioning of the multiapp would naturally match the partitioning of the file, and the transfers (for example multiappGeneralFieldShapeEvaluationTransfer) would handle the differences in partitioning

[robtclay]

Does GridPartitioner work for EBSDMeshGenerator? I tried implementing it similar to how it is implemented in "test/tests/meshgenerators/break_mesh_by_block_generator/hanging_nodes_parallel.i" example and it doesn't seem to do anything from what I can tell, and I still get the error.

Is there a way to use MultiApps without needing to rerun the first step every time? I was using SUO for this rather than MultiApps because I want to be able to run the 2nd step without needing to rerun everything that happens in step 1.

[GiudGiud]

nd it doesn't seem to do anything from what I can tell, and I still get the error.

Did you use it for both the input file generating the restart and the input using the restart file?
Did you get any output from the GridPartitioner mentioning it's ignoring your parameters?
Is your domain such that every cell is deterministically assigned to a grid cell? If the centroid lies on grid boundaries, it can be assigned either way.

Is there a way to use MultiApps without needing to rerun the first step every time? I was using SUO for this rather than MultiApps because I want to be able to run the 2nd step without needing to rerun everything that happens in step 1.

yes absolutely. Use the solutionUO in the multiapps, and use the same file for the mesh of the multiapps.
This loads the result in a distributed way.
Then the transfers handle the mapping from the multiapp mesh to the parent app mesh