Allow concurrent writers with Caterva2 - #703
Open
FrancescAlted wants to merge 9 commits into
Open
Conversation
Pre-sized uninit arrays written once per chunk, with the offsets block carrying the completion record. Measured what the format allows first: a chunk whose old content was special appends rather than compacting, so a first write costs ~0.5 ms and moves no other chunk's offset. Records why the frame length cannot serve as a validator (a zeros write can leave it unchanged) and why the .b2lock generation counter can, and why chunk writes do not generalise to fsspec/S3 the way the reads did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A pre-sized array is filled a chunk at a time, by as many writers as there are chunks: `C2Array.update_chunk` posts one, and the subscriber refuses a slot that already holds anything. `written_chunks` reads what the fill recorded in the frame's own offsets, so progress costs one range read and no endpoint of its own. Reading the index turned out not to be the same question as reading blocks of a chunk: `serves_blocks` also weighs whether splitting a chunk would pay, which is nothing to do with whether the frame has offsets to read. Split the geometry half out, so a frame of small chunks can still say which of them were written. `invalidate_index` drops what was read of a frame that has since been written to -- the header as well as the offsets, since a write moves the frame's length and its payload extent, and the offsets are found through both. Nothing is read until the next lookup asks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phases 1, 2, 3 and 5. Phase 4 stays open with what was learned about it written down: the ETag is the freshness half of the stamp question and does not answer the identity half, which is what tells an array that was appended to from one that was replaced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The stand-in accepts chunk writes now, so --write lays out an empty array of the dataset's geometry, fills it with the dataset's own chunks, and times the three things the design rests on: the fill serial and with several writers, what the server pays to store a chunk into an empty slot against over a live one, and what reading the progress costs from the offsets against walking the chunks. Two of those numbers came out wrong before they came out right, and both were the measurement's fault. Rewriting a live chunk with bytes of its own length is the case the frame skips the move for, so it has to carry a chunk of a different compressed size or it measures nothing; and the first FsspecNDSource of a process pays for fsspec's own import, which is 77 ms of nothing to do with reading offsets. The ratio a rewrite costs is the dataset's own -- it is whatever payload follows the chunk -- so the row says how many bytes were moved rather than leaving a bare multiple to be read as a constant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two questions wanted different answers, and mtime:cbytes was answering neither well. Which array this is now comes from the nonce a subscriber writes into vlmeta on the first chunk written: a size and an mtime can both be repeated by a different array at the same path, and a cache served against that one is wrong in every chunk without saying so. Whether it has changed since keeps the mtime and the size -- but only while the array is still being filled. A complete array is stamped by its nonce and size alone, since every slot is claimed and every write to it refused, so a cache of it stands where before an mtime that moved for reasons of its own threw the whole thing away. The plan wanted the stamp to hold still *during* a fill, on the grounds that a cache's chunks are all still where they were. They are not: a cache built while a chunk was unwritten holds the zeros an unwritten chunk reads as, and the run-length offset it had, and both are wrong the moment a writer fills that slot. A frozen stamp serves those zeros for good, which a test now pins by freezing one and watching it happen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review found three ways the write path could serve wrong data, and all three are fixed here with a test apiece. A handle reads api/info once when it opens the array, and `stamp` is built from exactly the fields a write moves -- so a handle used to fill an array kept its pre-fill stamp for good, and a Proxy given that handle adopted a cache built while the chunks were still empty. A write marks the metadata stale now, and whatever asks reads it again. `serves_blocks` weighs whether splitting a chunk into blocks would pay, which is nothing to do with whether the frame has an index worth reading -- but the two shared one memoized source, so touching the block path first (max_ranges is a public property, and Proxy reads it) cached None and left written_chunks raising NotRanged ever after. The judgement belongs at the call, not in the slot. A complete array stamps `n<nonce>:<cbytes>` and a filling one whose subscriber reports no mtime stamped the same string, which is the one collision the whole design exists to prevent. Both carry a marker now. Also from the review: invalidating the index no longer empties what index_state hands a cache, nor drops layouts a written chunk can never invalidate; _forget_index takes the lock a source is built under; the run-length tag is read in the host's byte order rather than always little-endian; and the fill bench built a C2Array per chunk rather than per writer, which charged every timed write an untallied api/info. The numbers move little -- 7.6x rather than 7.0x for eight writers over a simulated WAN -- and the plan is corrected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FrancescAlted
force-pushed
the
cat2-concurrent-writers
branch
from
August 21, 2026 08:11
cdd56d0 to
f4a4dc1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This implements the necessary changes to allow concurrent writers with Caterva2. Still preliminary.