evodb `curDBTransaction.IsClean()` abort from out-of-transaction credit-pool snapshot write at DISK_SNAPSHOT_PERIOD boundaries

[DashBot-0001]

Summary

dashd/dash-qt aborts at flush time with

dashd: evo/evodb.cpp:59: bool CEvoDB::CommitRootTransaction():
       Assertion `curDBTransaction.IsClean()' failed.
Posix Signal: Aborted

when a credit-pool lookup on the chain tip happens outside a dbTx while the
tip is on a DISK_SNAPSHOT_PERIOD (576) boundary. The interrupted evodb state
then fails VerifyDB on restart (*** Found EvoDB inconsistency, you must reindex to continue) even though the chain is at tip, so the node needs a full
-reindex to start again.

Reproduced deterministically on v23.1.8 (release). The relevant code
(CommitRootTransaction, CCreditPoolManager::AddToCache) is unchanged on
master at time of writing, so master is affected too.

Root cause

CCreditPoolManager::AddToCache() writes a snapshot to evodb unconditionally at
every 576-boundary, with no guarantee of being inside a committed transaction:

// evo/creditpool.cpp
void CCreditPoolManager::AddToCache(const uint256& block_hash, int height,
                                    const CCreditPool &pool) {
    { LOCK(cache_mutex); creditPoolCache.insert(block_hash, pool); }
    if (height % DISK_SNAPSHOT_PERIOD == 0) {
        evoDb.Write(std::make_pair(DB_CREDITPOOL_SNAPSHOT, block_hash), pool);
    }
}

evoDb.Write targets the cur layer, which is only committed/rolled back by the
CEvoDBScopedCommitter obtained from BeginTransaction(). AddToCache is
reached from ConstructCreditPool ← GetCreditPool, which has four call sites —
only one is inside a dbTx:

Call site	In a dbTx?
evo/specialtxman.cpp:573 GetCreditPool(pindex->pprev) (ConnectBlock)	yes (safe)
rpc/rawtransaction.cpp:693 GetCreditPool(pTipBlockIndex) (getassetunlockstatuses, no-height branch)	no
node/miner.cpp:471 GetCreditPool(pindexPrev) (block assembly)	no — but see note
node/interfaces.cpp:433 getCreditPoolCounts (Qt CreditPoolFeed, timer-driven)	no

A block's own credit-pool snapshot is written when its successor connects
(ConnectBlock uses pprev). So while the tip is a 576-boundary block, that
block's snapshot is not yet in evodb. An out-of-dbTx GetCreditPool(tip) then
misses both the in-memory cache and the snapshot, runs ConstructCreditPool(tip)
→ AddToCache(tip) → the bare evoDb.Write lands in the uncommitted cur
layer. Nothing commits or rolls it back, so the next FlushStateToDisk →
CommitRootTransaction() fails assert(curDBTransaction.IsClean()). If the next
block connects first, its ConnectBlock dbTx commits cur and the stray write is
harmless — hence this is intermittent rather than a guaranteed crash.

Reproduction (regtest, v23.1.8)

dashd -regtest -datadir=DIR -daemon=1 -fallbackfee=0.0001
dash-cli -regtest generatetoaddress 575 <addr>     # tip 575 (575 % 576 = 575)
dash-cli -regtest getassetunlockstatuses '["1"]'   # no height -> queries tip
dash-cli -regtest stop                             # clean shutdown

dash-cli -regtest generatetoaddress 576 <addr>     # tip 576 (576 % 576 = 0)
dash-cli -regtest getassetunlockstatuses '["1"]'   # no height -> queries tip
dash-cli -regtest stop                             # -> Assertion `IsClean()' failed

Observed matrix (V20 active): 575+call → clean; 576+call → abort;
576+no call → clean; tip 580 + getassetunlockstatuses '["1"]' 576 (explicit
historical boundary height) → clean; getblocktemplate at tip 576 → clean.

Notes on the other out-of-dbTx doors:

• getblocktemplate (miner.cpp:471) reaches the same bare write but did not
  abort in testing — its template-validity check appears to open and roll back a
  dbTx, whose RollbackCurTransaction() → Clear() incidentally wipes the stray
  write. This masking looks accidental.
• getCreditPoolCounts (interfaces.cpp:433) is polled by the Qt GUI's
  CreditPoolFeed on a timer, so a running dash-qt wallet is expected to hit
  this on its own at 576-boundaries with no user action. (Reasoned from the call
  path; not separately GUI-tested.)

Exploitability

We believe this is not remotely triggerable: the RPC door is credential-gated
and (in the configs we checked) bound to loopback/private ranges; Drive/Platform's
own get_asset_unlock_statuses always passes a chainlocked height, so the DAPI
path takes the safe historical-block branch; and there is no mempool/p2p path
into GetCreditPool. The GUI door is self-inflicted. It reads as a reliability
bug, not a remote DoS. If you consider node aborts security-sensitive per your
SECURITY.md, we're happy to move this to security@dash.org instead.

Possible fixes (for discussion, not prescriptive)

• Make the snapshot write in AddToCache conditional on being inside a committed
  transaction — read paths compute-and-cache only, durable writes go through the
  same committer ConnectBlock uses. (deterministicmns.cpp's snapshot writer
  only runs from ProcessBlock, always in a dbTx — a useful contrast.)
• Or wrap the out-of-dbTx GetCreditPool callers in a rolled-back
  BeginTransaction (read-only semantics) so any stray write is cleared — which
  is effectively what getblocktemplate does by accident today.

The assert itself is correct (it prevents writing an inconsistent evodb); the bug
is on the write side. Happy to test any candidate fix against the regtest repro.

[DashBot-0001]

A few additions from following this through to recovery, in case they help triage.

1. Field data point. On the node where we hit this, the last block written to
evodb before the abort was height 2517120, and 2517120 % 576 == 0 — i.e. the
tip was sitting exactly on a snapshot boundary, matching the mechanism above.
The abort came ~3 minutes after that block, while the tip was still on the
boundary (the next block had not yet connected).

2. Recovery is a full -reindex, and the lighter option is blocked.
-reindex-chainstate looks like the natural fix (block files and block index are
intact; only chainstate/evodb need rebuilding), but on a node with txindex it
errors out — and txindex is sticky from the presence of indexes/txindex/
even when it is not set in dash.conf or settings.json, so a node can be
txindex-enabled with no flag anywhere. Either -reindex-chainstate -txindex=0
(two-step) or a full -reindex works; we used the full reindex.

Worth noting because the error message says "you must reindex to continue"
without indicating which reindex, and the obvious lighter choice fails.

3. Amplification on containerised setups. With a restart: unless-stopped
policy, the single abort becomes a restart loop: each start re-runs VerifyDB,
fails the same way, and exits — in our case ~1400 times over several hours,
during which the node serves nothing and RPC returns -28. For a masternode
that ends in a PoSe ban, which then requires a separate ProUpServTx to clear
(resyncing does not lift the ban). So the blast radius of one stray write is
considerably larger than a single crash.

4. Re the Qt path (getCreditPoolCounts via CreditPoolFeed): in the
original report I flagged that as reasoned-from-call-path rather than tested.
Adding a supporting observation — we previously had a dash-qt wallet node and
a headless masternode hit this same assertion on the same day, with no
operator action on the wallet node. That is consistent with the GUI feed's timer
polling GetCreditPool(tip) on its own and self-triggering at a boundary,
though I still have not tested the GUI path in isolation.

5. Detection, for anyone else hitting this. The dangerous window is
predictable: it is the ~1 block interval whenever tip % 576 == 0, so the next
one is at ((tip / 576) + 1) * 576. That makes it straightforward to correlate
an unexplained evodb abort against the boundary schedule.

Happy to run any candidate patch against the regtest reproducer.

[sibTamias]

Field reports: same abort on evonodes at every 576-boundary (v23.1.8)

Confirming #7549 on dashmate evonodes (headless, no Qt). Several incidents, all on height % 576 == 0, same sequence as the issue: CommitRootTransaction IsClean() abort while tip sits on the boundary → Docker restart: unless-stopped → VerifyDB Found EvoDB inconsistency, you must reindex to continue loop → PoSe ban.

Environment

• Dash Core v23.1.8 (/Dash Core:23.1.8/, version 230108), image dashpay/dashd:23
• dashmate 4.1.1
• Ubuntu 22.04, Docker Compose stack (mainnet-core + Drive/tenderdash)
• Evonodes (not dash-qt). Core rpcwhitelist for drive_consensus includes getassetunlockstatuses.

---

1) 2026-06-24 — one evonode

• Tip 2493504 (2493504 % 576 == 0)
• CEvoDB::CommitRootTransaction(): Assertion curDBTransaction.IsClean() failed
• Then crash-loop: EvoDB inconsistency at 2493504
• PoSe ban followed. Full -reindex was started by mistake and aborted; recovered later from a consistent peer snapshot.

2) 2026-08-24 — several evonodes at once

Same second, same height — not disk corruption and not an operator stop/update.

UTC	What happened
00:18	Block 2526912 (2526912 % 576 == 0), DKG InitNewQuorum
00:22:39	CEvoDB::CommitRootTransaction(): Assertion curDBTransaction.IsClean() failed
00:23	Restart → VerifyDB → EvoDB inconsistency at 2526912 → crash-loop (RPC -28)

Other evonodes on the same Core version connected 2526912 without aborting — matches the issue’s “if the next block connects first, the stray cur write is committed/harmless” race.

PoSe bans on the crashed set. Recovered by rsync of Core+Platform datadir from a healthy node after dashmate stop --safe (not -reindex; txindex makes -reindex-chainstate unusable, as you noted).

A dashmate 4.1.1 package update happened after this crash and was not the trigger.

3) 2026-08-27 — one evonode again (after a successful 08-25 restore)

That node had been at tip for ~2 days after restore. Then:

2026-08-27T03:57:36Z CDeterministicMNManager::ProcessBlock -- Wrote snapshot. nHeight=2528640
2026-08-27T03:57:36Z UpdateTip: ... height=2528640 ... progress=1.000000
2026-08-27T03:57:36Z ActiveDKGSessionHandler::InitNewQuorum -- height[2528640] ...
dashd: evo/evodb.cpp:59: bool CEvoDB::CommitRootTransaction():
       Assertion `curDBTransaction.IsClean()' failed.
2026-08-27T04:00:36Z Loaded best chain: ... height=2528640
2026-08-27T04:00:37Z ERROR: VerifyDB(): *** irrecoverable inconsistency in block data at 2528640
                     hash=0000000000000011233904004bdde0e5e93f78710dc29d9214d484d338179513

• 2528640 % 576 == 0
• Abort ~3 minutes after the boundary block while still on that tip — same window as your field note on 2517120
• Docker restart loop for several hours; Masternode State = UNKNOWN; on-chain POSE_BANNED

Peers restored at the same time stayed up through this boundary — again consistent with an intermittent stray write, not a bad block.

---

Trigger on evonodes (vs Qt)

These nodes have no CreditPoolFeed / dash-qt. The remaining out-of-dbTx door that is actually wired on dashmate evonodes is Drive → Core RPC getassetunlockstatuses.

The issue text says Drive always passes a chainlocked height (safe historical branch). If that is true for every call, something else on the evonode path is still hitting GetCreditPool(tip) in this ~3 minute window (DKG / flush / helper). If Drive sometimes omits height, that would explain several evonodes in the same LLMQ dying together at 00:22:39 UTC on 08-24.

Happy to capture a debug.log around the next boundary (2529216 = 2528640 + 576) with RPC tracing if that helps.

Recovery note for operators

On Docker masternodes the abort is not “one crash”: VerifyDB on every start makes it a permanent loop and a PoSe ban. -reindex from genesis is not operationally acceptable (~2.5M blocks). A consistent copy of chainstate+evodb from a peer that did get the next block (so it never aborted) brings the node back; ProUpServTx is still required to clear the ban.

The assert(IsClean()) is doing its job; the durable snapshot write in CCreditPoolManager::AddToCache() at DISK_SNAPSHOT_PERIOD outside a dbTx is the bug. A fix in 23.1.x would stop this class of fleet outage.

[sibTamias]

Addendum — synchronous upgrade/restart and a shared race window

A group of evonodes was upgraded to Core 23.1.8 and restarted together: dashpay/dashd:23 core containers were created in the same ~40-second window, so they almost certainly came up on the same starting height (shared snapshot) and stayed in lockstep.

They later hit the same DISK_SNAPSHOT_PERIOD boundary at the same time, aborted together, and were PoSe-banned almost simultaneously.

Other evonodes on the same Core version that had been started a few minutes earlier or later survived that same block.

So this is still #7549. A synchronous upgrade/restart does not corrupt EvoDB by itself; it only puts that group into the same race window (same tip, likely the same Drive/helper timer phase) when height % 576 == 0.