Add vertical advection by brian-oneill · Pull Request #328 · E3SM-Project/Omega

brian-oneill · 2026-01-11T17:41:49Z

Add the VertAdv module to Omega, providing data structures and methods for vertical advection. This module includes:

Arrays for storing vertical velocity and tracer fluxes at layer interfaces.
A method to compute the vertical velocity from the divergence of the horizontal velocity.
Methods for computing the tendencies of thickness, horizontal velocity, and tracers due to vertical advection. The tracer tendency methods include multiple options for order of accuracy of the flux calculation, as well as algorithmic options for standard advection or flux-corrected transport (FCT).

Additional changes in this PR:

Extend parallelForOuter with an optional argument to enable the use of scratch arrays in hierarchical parallelism.
Add the auxiliary variable ProvThickness, which represents thickness after horizontal thickness flux. Further work required to fully implement provisional thickness.
Remove BottomDepth from HorzMesh, this field was previously added to VertCoord.

This PR builds and passes the unit tests successfully on Chrysalis, pm-cpu, pm-gpu, and Frontier (CPU & GPU).

Checklist

Fixes #338

mwarusz · 2026-01-15T19:18:08Z

When running VERTADV_TEST with bounds checking enabled I get out-of-bounds errors. The reason is VertCoord::init(false) gets NVertLayers from OmegaMesh.nc and this value is used to create the member arrays of VertAdv in VertAdv::init(). The value of NVertLayers in VertAdv and MaxLayerCell get overwritten later in the convergence test, but the VertAdv member arrays (e.g. VertFlux) don't get resized.

cbegeman · 2026-01-15T22:29:52Z

+
+                 for (int KVec = 0; KVec < KLen; ++KVec) {
+                    const I4 K      = KStart + KVec;
+                    const Real WAvg = 0.5_Real * (LocTotVertVelocity(Cell1, K) +


Do we want to use

Omega/components/omega/src/ocn/HorzOperators.h

Line 139 in f2e951a

InterpCellToEdge(const HorzMesh *Mesh);

so that we can easily switch from anisotropic to isotropic interpolation for all variables (here and elsewhere)?

We can’t use InterpCellToEdge here as is, since it takes an Array1DReal as input, while TotalVerticalVelocity is an Array2DReal. Supporting this would require either an overload or a templated version of the interpolator. It may make more sense to handle this outside this PR and identify any other places to implement this refactor.

brian-oneill · 2026-02-14T01:32:55Z

Polaris `omega_pr` suite

Baseline workdir: /lcrc/group/e3sm/ac.boneill/polaris/intelTest/baseline_omega_pr/
Baseline build: /lcrc/group/e3sm/ac.boneill/dev/intelCPU
PR build: /lcrc/group/e3sm/ac.boneill/myFork/intelBuild
PR workdir: /lcrc/group/e3sm/ac.boneill/polaris/intelTest/new_omega_pr
Machine: chrysalis
Partition: compute
Compiler: intel
Build type: Release
Log: /lcrc/group/e3sm/ac.boneill/polaris/intelTest/new_omega_pr/polaris_omega_pr.o1152821
Result: All tests passed

mark-petersen

Thanks for your documentation throughout. I fixed a few spelling errors.

mark-petersen · 2026-02-19T18:06:16Z

See compiled documentation here:

hyungyukang · 2026-02-19T21:25:00Z

Using my time-stepping working branch (https://github.com/hyungyukang/Omega/tree/hkang/omega/merge-vert-vel
), I tested this PR with the merry-go-round test in Polaris. Since the test does not currently support Omega directly, I made several adjustments to run it manually. I will document the procedure for running the test manually in a separate comment later.

As expected, I obtained a convergence rate of 1.971 with VerticalTracerFluxOrder: 2, which matches the convergence rate obtained in MPAS-Ocean:

hyungyukang · 2026-02-20T02:59:02Z

+   } else if (VAdv->VertAdvChoice == VertAdvOption::FCT) {
+      ThicknessForVAdv = AuxState->LayerThicknessAux.ProvThickness;
+   }
+   VAdv->computeTracerVAdvTend(TracerTend, TracerArray, ThicknessForVAdv,


Suggested change

VAdv->computeTracerVAdvTend(TracerTend, TracerArray, ThicknessForVAdv,

VAdv->computeTracerVAdvTend(LocTracerTend, TracerArray, ThicknessForVAdv,

hyungyukang

I’m approving this PR based on my testing and visual inspection. All the changes I requested have been addressed, and the code works as expected.

@brian-oneill, thank you very much for your hard work on this!

mark-petersen

Approving based on Hyun's convergence test above, and visual inspection. Successfully passed all cTests on perlmutter CPU and GPU with gnu compilers.

cbegeman

I ran the 5m merry-go-round case with Omega (above) and MPAS-Ocean (below) and the error patterns look very similar. The horizontal velocity for Omega is not yet available, as it relies on reconstruction to the x-direction.

cbegeman · 2026-02-22T22:58:44Z

Great work, @brian-oneill! I just wanted to check; has anyone yet verified the 3rd and 4th order convergence options? If not, I'll do this with the polaris merry-go-round test.

hyungyukang · 2026-02-23T15:54:21Z

Great work, @brian-oneill! I just wanted to check; has anyone yet verified the 3rd and 4th order convergence options? If not, I'll do this with the polaris merry-go-round test.

@cbegeman , I ran it last time with the 3rd convergence option. However, I was not able to obtain the 3rd order convergence rate for both Omega (1.981) and MPAS-Ocean (1.981). Could you please run it as well just in case I missed something?

cbegeman · 2026-02-23T22:22:32Z

Using one processor, I was able to reproduce @hyungyukang's results:

Testing whether this is fixed by #316

cbegeman · 2026-02-24T00:57:56Z

4th order advection with limiter (MPAS top, Omega bottom):

cbegeman · 2026-02-24T01:34:09Z

4th order advection without limiter (MPAS top, Omega bottom):

Also add TimeStep to Tendencies

sbrus89 · 2026-03-12T21:14:44Z

Passes all tests in omega pr suite on pm-cpu with baselines from current polaris main.

Polaris `omega_pr` suite

Baseline workdir: /pscratch/sd/s/sbrus/polaris_vert_advection_pr_suite_baseline/
Baseline build: /global/homes/s/sbrus/scratch/polaris_vert_advection_pr_suite_baseline/build
PR build: /global/homes/s/sbrus/scratch/polaris_vert_advection_pr_suite/build
PR workdir: /global/homes/s/sbrus/scratch/polaris_vert_advection_pr_suite
Machine: pm-cpu
Compiler: gnu
Build type: Release
Log: not found
Result: All tests passed

sbrus89 · 2026-03-12T22:14:44Z

Passes CTests on pm-cpu and pm-gpu with debug builds.

xylar · 2026-03-13T16:09:46Z

@sbrus89, we need to update the Omega submodule in Polaris with these changes, right? I'll need to update the submodule for #361 as well, so we can include both.

sbrus89 · 2026-03-16T15:18:57Z

@xylar, yes that's correct.

cbegeman reviewed Jan 15, 2026

View reviewed changes

Comment thread components/omega/configs/Default.yml Outdated

cbegeman reviewed Jan 15, 2026

View reviewed changes

Comment thread components/omega/src/ocn/VertAdv.cpp

hyungyukang reviewed Jan 25, 2026

View reviewed changes

Comment thread components/omega/src/ocn/VertAdv.h Outdated

hyungyukang reviewed Jan 26, 2026

View reviewed changes

Comment thread components/omega/src/ocn/VertAdv.cpp

brian-oneill marked this pull request as ready for review February 14, 2026 01:01

vanroekel requested a review from mark-petersen February 18, 2026 15:26

mark-petersen reviewed Feb 19, 2026

View reviewed changes

Comment thread components/omega/src/ocn/VertAdv.cpp Outdated

Comment thread components/omega/src/ocn/VertAdv.h Outdated

Comment thread components/omega/test/ocn/VertAdvTest.cpp Outdated

hyungyukang reviewed Feb 19, 2026

View reviewed changes

Comment thread components/omega/src/ocn/AuxiliaryState.h Outdated

Comment thread components/omega/src/ocn/AuxiliaryState.h Outdated

mark-petersen requested review from hyungyukang and mark-petersen February 19, 2026 20:23

hyungyukang reviewed Feb 19, 2026

View reviewed changes

Comment thread components/omega/doc/devGuide/VertAdv.md Outdated

cbegeman reviewed Feb 19, 2026

View reviewed changes

Comment thread components/omega/src/ocn/CustomTendencyTerms.cpp

cbegeman reviewed Feb 19, 2026

View reviewed changes

Comment thread components/omega/test/ocn/HorzMeshTest.cpp

cbegeman mentioned this pull request Feb 19, 2026

BottomDepth should not be read from the horizontal mesh #338

Closed

hyungyukang reviewed Feb 20, 2026

View reviewed changes

hyungyukang approved these changes Feb 20, 2026

View reviewed changes

mark-petersen approved these changes Feb 20, 2026

View reviewed changes

mark-petersen reviewed Feb 20, 2026

View reviewed changes

Comment thread components/omega/test/ocn/TendenciesTest.cpp Outdated

cbegeman reviewed Feb 22, 2026

View reviewed changes

cbegeman mentioned this pull request Feb 23, 2026

Standard non-monotonic higher order horizontal advection copied from mpas-ocean to Omega. #316

Merged

13 tasks

brian-oneill added 20 commits March 10, 2026 08:42

Add VertAdv to Tendencies

da4107d

Disable VertAdv tends in TimeStepperTest

16597a2

Add VertAdv to AuxState

a8853d1

Add computeVerticalVelocity to computeMomAux

d27bc79

Add VertAdv init and clear to OceanDriver methods

7584bfb

Add computeThicknessVAdvTend to computeThicknessTendenciesOnly

4828a49

Add computeVelocityVAdvTend to computeVelocityTendenciesOnly

005b96f

Add computeTracerVAdvTend to computeTracerTendenciesOnly

a92f9a6

Also add TimeStep to Tendencies

Make TimeStep nonoptional for computeTracerVAdvTend

36ec792

Add VertAdv devGuide and userGuide

6fa3aa0

Add early return to VertAdv methods when NVertLayers == 1

0a01a4c

Fix typos in comments and docs

9461d88

Update HorzMesh user doc

578a35f

Add sanity check for bottom depth

ad15ca6

Fix linting fail

f9485b4

Rename TracerTend to LocTracerTend in function call

5c2be5e

Remove commented-out line

9b49bf7

Rebase: remove TracerAux computeVarsOnEdge

5759e27

Rebase: fix variable name passed to computeVelocityVAdvTend

520bdf8

Rebase: fix getLayerThickness usage

b7fc515

brian-oneill force-pushed the omega/add-vertical-advection branch from fbf582f to b7fc515 Compare March 10, 2026 16:16

Exchange halo for BottomDepth

1815dba

sbrus89 merged commit 09b2f44 into E3SM-Project:develop Mar 12, 2026
1 check passed

xylar mentioned this pull request Mar 13, 2026

Merge E3SM/master into Omega/develop #361

Merged

6 tasks

brian-oneill mentioned this pull request Mar 16, 2026

Fixes Omega ssh calculation #335

Open

11 tasks

brian-oneill mentioned this pull request Apr 23, 2026

Remove the FluxTracerType option from configs/Default.yml #393

Merged

10 tasks

	VAdv->computeTracerVAdvTend(TracerTend, TracerArray, ThicknessForVAdv,
	VAdv->computeTracerVAdvTend(LocTracerTend, TracerArray, ThicknessForVAdv,

Conversation

brian-oneill commented Jan 11, 2026 • edited by xylar Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mwarusz commented Jan 15, 2026

Uh oh!

Uh oh!

cbegeman Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brian-oneill Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brian-oneill commented Feb 14, 2026

Polaris omega_pr suite

Uh oh!

mark-petersen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mark-petersen commented Feb 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hyungyukang commented Feb 19, 2026

Uh oh!

Uh oh!

Uh oh!

hyungyukang Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

hyungyukang left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mark-petersen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cbegeman left a comment

Choose a reason for hiding this comment

Uh oh!

cbegeman commented Feb 22, 2026

Uh oh!

hyungyukang commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cbegeman commented Feb 23, 2026

Uh oh!

cbegeman commented Feb 24, 2026

Uh oh!

cbegeman commented Feb 24, 2026

Uh oh!

sbrus89 commented Mar 12, 2026

Polaris omega_pr suite

Uh oh!

sbrus89 commented Mar 12, 2026

Uh oh!

Uh oh!

xylar commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sbrus89 commented Mar 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

brian-oneill commented Jan 11, 2026 •

edited by xylar

Loading

cbegeman Jan 15, 2026 •

edited

Loading

Polaris `omega_pr` suite

hyungyukang left a comment •

edited

Loading

hyungyukang commented Feb 23, 2026 •

edited

Loading

Polaris `omega_pr` suite

xylar commented Mar 13, 2026 •

edited

Loading