feat(csharp/src/Drivers/Databricks): CloudFetch Use RecyclableMemoryStream for LZ4 decompression to reduce memory pressure

[eric-wang-1990]

Summary

Implements RecyclableMemoryStream for LZ4 decompression to reduce memory pressure and fix stream corruption issues.

Before the change:

Method	ReadDelayMs	Mean	Min	Max	Median	Peak Memory (MB)	Gen0	Gen1	Gen2	Allocated
ExecuteLargeQuery	5	15.95 s	14.99 s	16.64 s	16.21 s	See previous console output	364000.0000	63000.0000	38000.0000	2.73 GB

For the dotNet Mem dump, the set_Capacity from MemoryStream is occupying 5GB.

After the change:

Method	ReadDelayMs	Mean	Min	Max	Median	Peak Memory (MB)	Gen0	Gen1	Gen2	Allocated
ExecuteLargeQuery	5	16.36 s	15.25 s	17.95 s	15.88 s	See previous console output	411000.0000	26000.0000	16000.0000	1.27 GB

For the dotNet Mem dump, we do not see memory allocation for set_Capacity.

CloudFetch

This only touches CloudFetch path for now.
For non-cloud fetch case where we use the syncrho version for Lz4 Decode, there will be another PR to address that if necessary.

Changes

• Modified Lz4Utilities.DecompressLz4Async to return MemoryStream instead of (buffer, length) tuple
• Stream is now disposed by the caller (Arrow) after reading, not immediately in the utility method
• Uses RecyclableMemoryStreamManager for pooled memory allocation
• Decompression uses CopyAsync with 80KB chunks for efficient streaming
• Fixed bug where buffer was recycled before Arrow finished reading it

Problem Fixed

Previously, the RecyclableMemoryStream was disposed immediately after decompression, returning its buffer to the pool. This caused "Unexpectedly reached end of stream" errors when Arrow tried to read from the buffer later, as it could have been overwritten by other operations.

Solution

The new API returns the stream itself without disposing it. Arrow reads from the stream and then disposes it when done, ensuring the buffer is only returned to the pool after reading completes.

Benefits

• ✅ Reduces GC pressure by reusing memory streams from pool
• ✅ Fixes "Unexpectedly reached end of stream" error
• ✅ Memory is only returned to pool after Arrow completes reading
• ✅ More efficient memory usage for high-volume CloudFetch operations
• ✅ Uses CopyAsync for chunk-by-chunk decompression (80KB chunks)

Testing

• Code compiles successfully on Mac for all target frameworks (netstandard2.0, net472, net8.0)
• Needs testing on Windows with benchmark suite
• Verify CloudFetch E2E benchmark completes without stream errors

🤖 Generated with Claude Code

[eric-wang-1990]

@CurtHagenlocher This is ready to merge, please let me know if you have any concerns

[CurtHagenlocher]

Thanks, this looks really promising! I have concerns about the lifetime of the RecyclableMemoryStreamManager, especially since no limit is set on the pool size. Perhaps it can be associated with the lifetime of the driver, database or connection?

[CurtHagenlocher]

Looks like this will need rebasing.

[eric-wang-1990]

@CurtHagenlocher The new result with both commits further reduce the memory to 541MB from 2.73GB.

Method	ReadDelayMs	Mean	Min	Max	Median	Peak Memory (MB)	Total Rows	Total Batches	Gen0	Gen1	Gen2	Allocated
ExecuteLargeQuery	5	12.45 s	10.43 s	14.46 s	12.46 s	284.13	1,439,935	743	385000.0000	33000.0000	6000.0000	541.65 MB