Garnet is a remote cache-store from Microsoft Research that offers strong performance (throughput and latency), scalability, storage, recovery, cluster sharding, key migration, and replication features. Garnet can work with existing Redis clients.
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Suppress CredScan finding for testcert.key.pem test certificate (#2074) The PEM certificate support added in #1937 introduced test/testcerts/testcert.key.pem, a PEM-encoded private key for the self-signed certificate used by TLS unit tests. CredScan flags its private key, breaking the compliance build. Add it to the CredScan exclusion list alongside the other test certificate/key files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ebda3282-4ac2-49f9-820a-d658443b15ae | 19 天前 | |
[Storage] Optimize IOPS for RAID-0 NVMe disks (#2018) * Optimize NativeStorageDevice random-read IOPS (4.14M -> 6.94M on 8x NVMe RAID-0) WIP: raise Garnet SSD random-read serving throughput toward the fio ceiling (8.24M IOPS). Verified on KV.benchmark scenario 2 (100M x 100B, log-mem 16m, 100% random 4K reads from disk, native libaio), standalone runs, node0-pinned. Fixes in this branch: - FIX#1 (NativeStorageDevice.cs): shard per-submitter in-flight tracking; removed the global numPending counter + freeResults queue that cache-line ping-ponged across all submit/complete threads. Positive thread scaling. - FIX#2 (Device.benchmark/BenchWorker.cs): drop the per-op device.TryComplete() from the submit hot path (serialized all submitters on ctx0's kernel ring_lock); dedicated drainers do all reaping. - FIX#3 (cc/device/thread.h, thread_manual.cc): cache-line-pad Thread::id_used_[] (per-IO EpochGuard acquire/release CAS was false-sharing). Native .so rebuilt. - FIX#4 (NativeStorageDevice.cs): per-shard free-list for completion slots. The prior counter-ring slot reuse was unsafe under OUT-OF-ORDER device completion (a slow IO's slot could be overwritten by newer submits wrapping the ring), corrupting the AsyncIOContext and crashing KV. Slots now return to a free-list only after their own IO completes. - FIX#5 (Utilities/BufferPool.cs): stripe SectorAlignedBufferPool's per-level free-list across 128 sub-queues, thread-affine. The single ConcurrentQueue per size-level was 52.6% of all CPU (TryDequeue+TryEnqueue) under the pending-read workload. 4.14M -> 6.30M @32thr. General win for all Garnet disk reads. - FIX#6 (NativeStorageDevice.cs, Devices.cs, benchmarks): decouple num_io_contexts from num drainer threads via new numIoContexts option (--device-io-contexts / --io-contexts). Default 0 == legacy 1:1 (byte-for-byte unchanged). Multi-ring drainers range-POLL their rings (never block on one ring, which would starve the siblings). Helps low/medium thread counts; neutral at peak. Peak verified: 6.94M ops/s @48 threads (84% of fio). All 89 DeviceTests pass. TODO (not in this commit): batched io_submit; io_uring IOPOLL + registered buffers/files; patchelf .so libaio.so.1t64 -> libaio.so.1 + libaio-only variant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add opt-in device tuning levers: batched libaio submit + affine inline drain Two additional, opt-in (default = legacy behavior) levers explored while closing the KV random-read gap on 8x NVMe RAID-0. Both were measured neutral at the core-saturated peak (~6.9M) but positive at lower/medium thread counts, and are kept behind env flags with zero-regression defaults. FIX#7 batched libaio submit (GARNET_SUBMIT_BATCH, default 1 = immediate submit): - Native (file_linux.cc/.h, native_device.h, native_device_wrapper.cc): per-submitter thread_local accumulation of prepared READ iocbs, flushed via io_submit(ctx, N) at a threshold; handles partial submit / EAGAIN / per-iocb permanent error. New NativeDevice_FlushSubmits C ABI (uring backend = no-op). - Managed (NativeStorageDevice.cs, IDevice.cs, StorageDeviceBase.cs): FlushSubmits() + P/Invoke; TryComplete() flushes the calling thread's batch first (throttle-spin safety net). KV benchmark (KvBenchmark.Worker.cs) flushes the sub-threshold tail per read batch. Affine inline drain (GARNET_INLINE_DRAIN_AFFINE, default off): - New IDevice.TryCompleteMine() (StorageDeviceBase falls back to TryComplete; native NativeStorageDevice drains only the caller's affine context/ring). The inline submitter-thread completion path (TsavoriteThread.cs, AllocatorBase.cs) uses it when the flag is set, cutting per-context io_getevents syscalls at lower thread counts. Rebuilt both prebuilt Linux natives from current source (USE_URING=ON -> libnative_device.so, USE_URING=OFF -> libnative_device_libaio.so) so both export the new NativeDevice_FlushSubmits / NativeDevice_TryCompleteMine entrypoints; TryComplete calls FlushSubmits unconditionally, so the libaio-only fallback must have them too. Tsavorite build clean (0 warnings); 89 DeviceTests pass; default path is byte-for-byte legacy behavior (both env flags unset). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Resp.benchmark data races that crashed multi-threaded loads Two concurrency bugs made the offline benchmark crash (NullReferenceException in ReqGen.GetRequestArgs) and produce malformed requests under load: 1. Shared request list mutated in place. GetRequestArgs() returns a reference to a cached List<string> owned by ReqGen. GarnetClientSessionOperateThreadRunner did reqArgs.Insert(0, "MSET") on it every iteration. That both (a) raced across threads (concurrent List.Insert corrupts the backing array) and (b) permanently prepended "MSET" to the shared list, growing it unboundedly even single-threaded. Fix: build a fresh args array with the command prepended; never mutate the cache. 2. Non-thread-safe System.Random for serve-offset selection. GetRequest() and GetRequestArgs() called a shared System.Random ('r') concurrently from all worker threads, which can return out-of-range indices. Fix: use Random.Shared (thread-safe) for the concurrent serve-offset draw. 'r' is retained for the single-threaded generation phase where its deterministic seed matters. Also add --load-threads (default 8) to parallelize the initial data-load phase. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make affine inline device drain the default (fix RESP read-throughput degradation) The inline submitter-thread completion drain (Tsavorite CompletePending / AsyncGetFromDisk throttle-wait) is the primary reaper for disk-bound reads. The legacy inline drain, IDevice.TryComplete(), reaps only a single fixed io_context (context 0), so every inline-draining thread serialized on that one context's kernel aio mutex. On the RESP GarnetServer, disk-read completions are processed by the .NET thread pool, which grows under disk latency. With many workers all draining context 0, ~46% of server CPU was spent spinning on the aio mutex (osq_lock), and 100% random-read throughput spiralled downward run-over-run (6.75M -> 2.4M ops/s on an 8x NVMe RAID-0) as the pool grew. The fixed-context drain also only covered context 0's 1/N share of completions inline. IDevice.TryCompleteMine() reaps the calling thread's own affine context (the one its submits land on), spreading the inline drain across all contexts. This removes the mutex storm (osq_lock 46% -> 7%) and holds throughput stable at ~7.2M ops/s across many runs with no degradation. It was previously measured neutral at the uncontended saturated peak, so making it the default is a strict improvement. Devices that do not shard completions fall back to TryComplete() automatically, so this is safe for all device types. Flip Constants.InlineDrainAffine to default on; set GARNET_INLINE_DRAIN_AFFINE=0 to restore the legacy fixed-context-0 drain. 89 DeviceTests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add opt-in io_uring batched submit (device-level +14%, uring reaches libaio parity) Defer io_uring_submit to coalesce many read SQEs into one submit syscall, mirroring the existing libaio batch. Opt-in via GARNET_SUBMIT_BATCH (default 1 = submit per-op = byte-for-byte legacy, zero regression). Only a thread that solely owns its ring defers (per-ring CAS ownership); ring-sharing threads (submitters > rings) fall back to per-op submit. Writes never batched. Every deferred SQE carries its io_context as user_data, so exactly one completion is dispatched regardless of which thread flushes. The managed TryComplete/TryCompleteMine (and the AsyncGetFromDisk throttle-spin) already call NativeDevice_FlushSubmits before draining, upholding the flush-before-wait invariant (no throttle deadlock) with no managed changes. get_sqe==null flushes the pending batch before retry/unwind; a real UringIoHandler::FlushSubmits with -EBUSY drain-assist (TryCompleteFor) replaces the previous no-op stub. Device.benchmark (512B random reads, t=32, io-contexts=32, no pin): uring 7.85M -> 8.94M at batch=32 (+14%), a new uring high reaching ~parity with libaio-batched (9.23M); integrity verified (71.5M ok == submitted, 0 err). Confirms per-op submit was uring's disadvantage vs libaio's batched io_submit. RESP serving is unchanged (closed-loop, managed-CPU-bound; batching neutral). BenchWorker.cs: flush the deferred tail batch via TryComplete in the shutdown drain so a sub-threshold tail never strands the exit. Native .so rebuilt (uring + libaio variants; libaio path is functionally unchanged). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add GARNET_DEVICE_IO_CONTEXTS override to decouple io_uring rings from drainers Give each submitter thread its own ring (rings >= completion threads) to avoid the shared-ring non-owner submit path, independently of the drainer count. The value is clamped up to --device-completion-threads so every drainer owns at least one ring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update device/KV/RESP benchmark READMEs with current run instructions Document the device tuning flags (--device-io-contexts, --device-completion-threads, --device-throttle-limit, --device-io-backend) and the record/page/segment settings used to reproduce the current SSD random-read numbers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Batch-reap io_uring completions in UringIoHandler::TryCompleteMine The inline submitter-thread completion path (Tsavorite CompletePending / AsyncGetFromDisk throttle-wait) drains the caller's own affine ring via TryCompleteMine, which reaped exactly one CQE per call (TryCompleteFor -> io_uring_peek_cqe). The dedicated drainer's QueueRunFor already batch-reaps up to kCqeBatch CQEs per cq_lock section, and libaio's QueueIoHandler::TryCompleteMine already batches via io_getevents; the uring inline path did not, so the network thread paid one cq_lock + one dispatch loop iteration per completion on the hot critical path. Add UringIoHandler::TryCompleteMineBatch: a non-blocking single pass that reaps up to kCqeBatch (64) completions in one cq_lock section (io_uring_peek_batch_cqe -> snapshot -> io_uring_cq_advance -> release -> dispatch outside the lock), mirroring QueueRunFor's phase-2. TryCompleteMine now calls it. One managed NativeDevice_TryCompleteMine P/Invoke therefore delivers a batch of completions, which the same thread drains inline. On disk-served RESP GET (100M x 128B, uring, 96 rings, 2 drainers) this raises throughput about 20% at the t=48 peak (5.94M -> ~7.1M ops/s) and removes the prior drainer-count sensitivity (the network threads now self-drain in batches, so a single background drainer no longer collapses under throttle-spin). The libaio backend is unaffected (its TryCompleteMine already batches); the UringIoHandler code is compiled only under FASTER_URING, so the libaio-only prebuilt is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add GARNET_URING_BATCH_REAP opt-out gate for the TryCompleteMine batch-reap Make the io_uring completion batch-reap (TryCompleteMineBatch) an opt-out knob so its throughput impact can be measured on the same binary without a revert-build. Default ON (unchanged behavior); GARNET_URING_BATCH_REAP=0 falls back to the legacy single-CQE reap (TryCompleteFor). Read once via a function-local static. Consistent with the other env-gated device levers (GARNET_DEVICE_IO_CONTEXTS, GARNET_SUBMIT_BATCH). Same-build ablation shows batch-reap is neutral at the saturated peak (a harmless CPU efficiency, not a peak-throughput lever), so this gate documents that finding and keeps it toggleable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix NativeStorageDevice throttle-divisor runaway that slowly starves device queue depth The per-submitter-thread in-flight sharding split the global ThrottleLimit into a per-thread budget via PerThreadLimit() = ThrottleLimit / activeShards. activeShards was Interlocked.Increment-ed the first time each thread submitted (AssignShard) but never decremented. Under the .NET ThreadPool, submitter threads are transient: they retire when idle and fresh ones are injected on the next burst. Each fresh thread bumped activeShards, so over a long-lived device the divisor ratcheted up without bound even though the number of concurrently active submitters stayed roughly constant. As a result the per-thread in-flight budget collapsed over the process's lifetime (e.g. 4096/35 -> 4096/245), so the aggregate device queue depth starved from ~ThrottleLimit down to a small fraction of it, and disk-serving read throughput declined progressively (observed ~55% over successive RESP GET runs), recoverable only by restarting the server. In-memory workloads were unaffected because they never exercise the pending-read throttle path. Fix: keep AssignShard's immediate increment (new threads instantly get a fair share) but periodically reconcile activeShards DOWN to actual shard occupancy (shards with in-flight > 0, ~= concurrently active submitters) via MaybeReconcileActiveShards(), time-gated to at most once per 200 ms by a cheap non-atomic tick check on the completion path. Births are counted immediately; deaths are reclaimed lazily from live occupancy. This only resizes the throttle divisor (a perf knob) and does not touch slot allocation, the submitted/completed balance, or completion routing, so it cannot affect correctness. Validated: RESP disk GET is now flat across successive idle-gap-separated runs (was a monotonic ~55% collapse); 89 DeviceTests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add opt-in LightEpoch-style io_uring ring affinity (GARNET_RING_LE_AFFINITY) The committed uring model owns a ring per submitter thread via a sticky `ring_owner_[idx]` that is released only at teardown. On .NET's oversubscribed ThreadPool this orphans rings: a retired thread leaves its tid in the owner slot forever, so that ring can never batch-own again and degrades to per-op submit ("dead-tid poisoning"). This adds an opt-in ownership model that maps LightEpoch's thread-affinity pattern onto ring ownership, env-gated by GARNET_RING_LE_AFFINITY (default off = byte-identical legacy path): - thread-local preferred ring (== LightEpoch startOffset1), re-acquired warm each network batch; - probe-and-replace on collision (== TryAcquireEntry circling the table): CAS-claim a free ring and adopt it as the new preferred; - release at the network-batch boundary (== Release-on-Suspend): the reliable "suspend" point after which the thread will not submit again until its next batch, so a churned-away thread's ring is reclaimed instead of orphaned. The batch-boundary release is gated on actual disk-read activity: a `[ThreadStatic]` flag is set in NativeStorageDevice.ReadAsync (the single choke point every disk read funnels through; pure in-memory hits never reach it) and consumed by EndBatchReleaseRing() in RespServerSession's batch finally. A batch that touched no disk pays one thread-static check and no P/Invoke, so the feature is free on in-memory and mixed workloads. Native: file_linux.{h,cc} add le_affinity_enabled() (cached), pick_ring_index_le() (warm fast-path / CAS-claim / probe-adopt), release_my_ring(), and a unified uring_thread_id() so pick and submit agree on the owner id; libaio's QueueIoHandler gets a no-op release_my_ring(). native_device.{h} + native_device_wrapper.cc export NativeDevice_ReleaseRing. AllocatorBase / LogAccessor expose the log IDevice so the session can reach the device. Measured (pinned, uring, 96 rings, submit-batch 32, fresh 100M keylen16 val96, OFF/ON/OFF bracket): throughput-neutral vs the committed model on both 100% random disk read (t=32 ~6.99M, t=48 ~7.9M; ON bracketed by OFF) and in-memory (t=32 ~52M, t=48 ~85M). 89 DeviceTests pass; GarnetServer builds warning-free. RESP disk serving is managed-CPU-bound, so this is a cleaner, non-poisoning ownership model rather than a throughput lever; kept opt-in pending a broader workload matrix (libaio, mixed, churn soak). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Windows ThreadPoolIoHandler no-ops for new NativeDeviceImpl forwards The branch's native device changes added three methods that NativeDeviceImpl<H> forwards to its handler_ (TryCompleteMine, FlushSubmits, and release_my_ring via ReleaseRing). These were only implemented on the Linux libaio/io_uring handlers. The Windows default (NativeDeviceImpl<ThreadPoolIoHandler>) failed to compile under MSVC once the native-build workflow exercised the Windows RIDs: error C2039: 'release_my_ring' is not a member of 'ThreadPoolIoHandler' Add the three as no-ops on ThreadPoolIoHandler, matching the existing Windows IOCP no-op style: completions fire on threadpool threads (no caller-affine inline drain), submits are immediate (no batch to flush), and io_uring ring ownership is Linux-only (nothing to release). Linux-only file, no effect on the Linux build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 30764320573) from the native sources on 'badrishc/optimize-device-iops'. * Document device completion model + high-latency (cloud) tuning Add a 'Completion model & high-latency (cloud) devices' section to the Device.benchmark README clarifying that the disk-read completion path is block-on-signal (WaitPending parks on the readyResponses SemaphoreSlim; the drainer parks in io_getevents/io_uring_wait_cqe_timeout with io_uring flags=0, i.e. no SQPOLL), not busy-spin. Documents the two poll levers (GARNET_INLINE_DRAIN_AFFINE inline peek; submit-side throttle backpressure) as saturation-only optimizations that fall through to the block path on high-latency devices, plus the cloud tuning knobs (disable inline affine drain; size --throttle-limit to the bandwidth-delay product). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Fix GARNET_SUBMIT_BATCH>=2 read-then-block deadlocks (recover + scan) The opt-in batched-submit device backend (GARNET_SUBMIT_BATCH) defers io_submit until a submitter thread's per-thread read batch reaches its threshold. Any path that issues a sub-threshold burst of device reads and then blocks waiting for their completions without flushing the batch deadlocks: the reads are never submitted, so the completions never fire (drainers idle in io_getevents while the issuer blocks on a never-signaled countdown/semaphore/readyResponses). Two sub-cases, both fixed by flushing the issuing thread's batch: 1. Read-issue-then-block sites (recovery, index, AOF/log scan frames). Flush via IDevice.TryComplete() (flushes before draining on NativeStorageDevice; no-op on immediate-submit devices) right after each read-issue loop, on the issuing thread (the batch is thread-local; async waits may resume on a different thread, so flush at the issue site, not the wait site): - IndexRecovery.BeginMainIndexRecovery (hash-table read; runs first) - MallocFixedPageSize.BeginRecovery (overflow-bucket read) - DeviceLogCommitCheckpointManager.ReadInto (checkpoint metadata read) - AllocatorBase.AsyncReadPagesForRecovery (hybrid-log pages) - AllocatorBase.AsyncReadPageFromDeviceToFrame (scan/iterator frames) - TsavoriteLogAllocatorImpl.AsyncReadPageFromDeviceToFrame (AOF/log scan frames) - ObjectAllocatorImpl object-log read-back sites (truncate, partial-sector flush) The index hash-table read runs first and is a single chunk for a 1g index, so the --recover hang is guaranteed for any batch >= 2. 2. Read re-issued from INSIDE a completion callback (disk hash-chain walk in AllocatorBase.TryVerifyOrReissuePendingRead, reached by the scan-cursor path ScanLookup -> CompletePending(wait:true) that backs RESP SCAN over disk, and by any pending read whose key mismatches on a hash collision). This runs on the completion/drainer thread (or an inline drainer) which returns to a blocking wait without flushing; the re-issued read strands in that thread's sub-threshold batch. CompletePending(wait:true) does flush, but the re-issue happens during its drain (after the flush) and then WaitPending blocks before the next flush -- so the outer flush does not cover it. Fixed with a new flush-ONLY IDevice.FlushSubmits() primitive (submit without draining), called right after the re-issue. Flush-only (not TryComplete) avoids re-entering the completion path from within a completion callback (no recursion on a long chain). Default virtual no-op on StorageDeviceBase; overridden on NativeStorageDevice; no-op on immediate-submit devices. Managed-only; no native change (works with the committed prebuilt .so). Verified: SpanByteIterationPendingCollisionTest (scan-cursor chain-walk) hangs under GARNET_SUBMIT_BATCH>=8 without the fix and passes (~160ms, same as unset) with it across batch {8,32,64}; full SpanByteLogScanTests (10) pass under batch=8 (previously hung the whole process ~75-100s); component recovery tests hang under batch=8 without the recovery flushes and pass with them; full-server --recover reaches Ready across batch {none,8,32,64,1024} and recovers 4.95M keys correctly. Full recovery suite (197) and 89 DeviceTests pass; scan tests (30) pass under batch-unset; GarnetServer Release builds 0-warn; default (non-batching) path is an unchanged no-op poll. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rationalize Native device IO tuning knobs; remove submit batching Distills the device-IOPS optimization study into a small first-class GarnetServer tuning surface and removes the opt-in submit-batching machinery, which was RESP-marginal and the sole source of io_uring ring-ownership ("dead-tid") complexity. Knob surface (all --device-*, wired identically into GarnetServer, KV.benchmark, and Device.benchmark): - Promote io-contexts (ring count) from the GARNET_DEVICE_IO_CONTEXTS env var to a real --device-io-contexts CLI option, plumbed through the LocalStorageNamedDeviceFactory chain to CreateLogDevice. This is the critical io_uring lever (too few rings serialize submitters on a per-ring lock, ~3x slower); libaio is indifferent. - Add --device-queue-depth (per-ring kernel submission depth) and split the old throttle double-duty cleanly into three orthogonal knobs: io-contexts = ring count, queue-depth = per-ring depth, throttle-limit = aggregate in-flight backpressure (<= io-contexts * queue-depth). The hidden min(_,4096) throttle clamp and the throttle/rings ring-depth derivation are removed. Freeze winning behaviors as permanent defaults (delete the env gates): - Affine inline device drain (was GARNET_INLINE_DRAIN_AFFINE) - io_uring batch-reap of completions (was GARNET_URING_BATCH_REAP) - libaio batched io_getevents completion (was GARNET_TRYCOMPLETE_BATCH) Remove entirely: - GARNET_SUBMIT_BATCH + the libaio/io_uring deferred-submit machinery - GARNET_RING_LE_AFFINITY + try_own_ring / release_my_ring / pick_ring_index_le / ring_owner_ ring-ownership state - IDevice.FlushSubmits and the seven batching-driven flush calls injected into the Tsavorite recovery/scan read-then-block paths; those paths revert byte-identical to origin/main, removing the whole latent-deadlock class along with the feature that motivated it. Retains exactly one intentional IDevice addition, TryCompleteMine() (affine inline drain; default = TryComplete(); fixes the context-0 osq_lock storm), and one inert-by-default capacity pair (numIoContexts + queueDepth, default 0 = legacy behavior). Docs: document the 4-knob surface + io-contexts guidance on the config page; canonicalize benchmark option names and refresh the READMEs. 89 DeviceTests pass; GarnetServer Release builds 0-warn; libaio RESP t=48 reqb=1024 reproduces the batch-free floor (7.28M median). Native binaries: linux-x64 rebuilt locally; the remaining RIDs are regenerated by native-build.yml on dispatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 30872979357) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] Right-size NativeStorageDevice NumShards 512 -> 128 Performance-driven right-sizing of the internal per-submitter-thread in-flight sharding constant. A sweep over NumShards {512..16} x threads {32..128} on both Device.benchmark (submitters == threads, the crisp knee) and RESP GET (real GarnetServer, ThreadPool submitters ~35-70) shows throughput is flat down to NumShards ~= the peak concurrent submitter count, with a knee only at NumShards ~= threads/4 (>=4 max-in-flight submitters share one 256-slot free-list => RentSlot spin). 128 gives ~2x headroom over the observed peak submitter count (~70) while trimming the fixed managed slot table from ~4.1 MB to ~1 MB. Every batch-free performance floor is reproduced or exceeded at NumShards=128 (median-of-3, pinned): device 512B libaio 8.82M / uring 8.61M; RESP GET libaio t48 7.41M / t64 7.45M, uring t48 7.41M. 89 DeviceTests pass. Managed-only; no native change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Derive NativeStorageDevice NumShards from ProcessorCount The in-flight shard count must stay >= the peak concurrent submitter count to keep free-list de-contention flat: a knee appears only at NumShards ~ submitters/4, where >=4 max-in-flight submitters share one 256-slot free-list and RentSlot spins. That peak is bounded by the logical processor count, so size NumShards = clamp(2 * ProcessorCount, 128, 1024) instead of a fixed 128 fitted to one machine: - floor 128: the value validated on the sweep hardware (no regression on smaller boxes; ~1 MB fixed managed memory). - 2x ProcessorCount: headroom for transient ThreadPool overshoot under connections >> cores. - cap 1024: bounds the fixed table to ~8 MB on very large machines. Environment.ProcessorCount honors process CPU affinity and cgroup limits, so pinned or containerized servers size to their usable cores. Managed-only; every NumShards use is modulo / loop bound / heap-array size (AssignShard already uses % NumShards), so no compile-time const is required. Builds 0-warn; dotnet format clean; 89 DeviceTests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Harden NativeStorageDevice: fix GPT/Gemini review findings Address cross-environment robustness issues found by the largest GPT (gpt-5.6-sol) and Gemini (gemini-3.1-pro-preview) model reviews of the device-IOPS optimization branch: - Throttle default (highest value): the ctor set ThrottleLimit = 120 (copied from the managed in-box devices), which silently defeated the intended DefaultThrottleLimit (4096). With ~50 active shards that capped out-of-box per-thread in-flight at ~2, throttling the device to ~4-5x below its peak unless the operator passed --device-throttle-limit explicitly. Set the ctor default to DefaultThrottleLimit so the `ThrottleLimit > 0 ? ... : DefaultThrottleLimit` fallback in PerThreadLimit()/init resolves 4096 out of the box. Validated: an out-of-box RESP GET server (no throttle flag) now sustains the ~7.5M libaio peak (t48, reqb1024), matching the explicit-4096 arm. - Dual libaio/liburing repair: extract LoadWithLibaioShim() and route BOTH the primary (Uring) and libaio-only fallback loads through it, so a host that needs the libaio SONAME shim (Ubuntu 24.04+ libaio.so.1t64) AND the liburing2 fallback is repaired regardless of which unresolved SONAME the dynamic loader reports first (previously one ordering dead-ended). - Results-slot clear: ReturnSlot() now clears results[offset] before re-enqueuing the slot, so a completed IO's captured callback delegate and context object are not kept rooted until the slot is next rented (bounded by MaxResults on a mostly-idle device). Cleared before the enqueue so a concurrent RentSlot cannot be clobbered. - ABI probe: the startup native-export probe now also calls NativeDevice_TryCompleteMine so a stale prebuilt .so missing the affine inline-drain export fails fast with the rebuild instruction (it is on the default hot path) instead of an EntryPointNotFoundException mid-run. - AssignShard int-overflow: reduce the round-robin counter modulo NumShards as uint so a long-lived thread-churning server that wraps nextShardSeq past int.MaxValue cannot produce a negative shard index. - Dispose contract doc: document that Dispose() must not be called from inside an IO completion callback (the inline affine-drain path cannot be cheaply detected on the hot completion path), matching the IDevice lifecycle contract. Also clarify the --device-throttle-limit help text and defaults.conf comment (0 => 4096 for the Native device, 120 for the managed in-box devices). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [CI] native-build: verify all load-bearing device exports on every RID The C# NativeStorageDevice loader now hard-probes NativeDevice_NumIoContexts, NativeDevice_QueueRunFor and NativeDevice_TryCompleteMine at device creation (the startup ABI probe; the affine inline-drain path that calls TryCompleteMine is on by default), in addition to NativeDevice_CreateWithBackend bound by the import resolver. A prebuilt binary missing any of these throws at server startup on that RID. The native-build workflow only verified CreateWithBackend, and only on Windows (the Linux job had no export check at all), so a native refactor that dropped one of the now-required exports would pass CI and only fail at runtime on some RID. Add an export-verification step to the Linux job (host nm reads the ELF dynamic symbol table for x64/arm64 and glibc/musl alike) and expand the Windows check to all four symbols. Build-only; does not touch the checked-in binaries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Test] Fix flaky PrimaryUnavailableRecoveryAsync CLUSTERDOWN race The PrimaryUnavailableRecoveryAsync(*, False) cluster tests intermittently failed in CI with "CLUSTERDOWN Hash slot not served" in place of the expected value. During CLUSTER FAILOVER FORCE a replica's role flips to "master" (TryTakeOverForPrimary assigns slots and role) before the failover session clears its recovery flag (EndRecovery). While the promoted primary is still recovering, reads to its own slots are answered with CLUSTERDOWN (ClusterSlotVerify). The test helper UpgradeReplicasAsync only waited for role == master, then immediately issued GETs that raced the still-open recovery window. Wait for LAST_FAILOVER_STATE == "failover-completed" on both promoted replicas after the role flip; that state is set only after the failover session (including EndRecovery) fully returns, so the following reads no longer race recovery. Add a WaitForFailoverCompleted overload keyed by IPEndPoint; the existing int overload delegates to it (behavior-preserving for its callers). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Cap default libaio io_setup reservation to fit stock fs.aio-max-nr libaio's io_setup permanently reserves io-contexts * queue-depth events from the GLOBAL fs.aio-max-nr budget at device creation, whether used or not. The DefaultQueueDepth ceiling (4096, correct for io_uring's per-ring mmap SQ) over-reserves for libaio: a single-ring auxiliary device reserves 4096 events it can never use deeply. In a multi-node cluster process ~15 such auxiliary devices (per-node AOF append, checkpoint bulk IO, replication logs — all default to a single ring) coexist, reserving ~15*4096 = 61440 ≈ a stock 65536 budget, so a transient overlap fails io_setup with errno 11 (EAGAIN). This surfaced as a flaky ClusterFailoverAttachReplicas CI failure ("Native device initialization failed: ... errno 11"). Fix: when --device-queue-depth is left at the default, size the libaio io_setup reservation to the throttle share instead of the io_uring ceiling — NextPow2(2 * ceil(throttle / io-contexts)), floored at 128, capped per-ring at LibaioReservationCap=2048. Deep in-flight should come from MORE rings (higher --device-completion-threads), not one mega-deep ring a lone drainer cannot keep saturated. Multi-ring serving devices (io-contexts >= 4) are unaffected: their 2x throttle share is already <= the cap, so io-contexts * reservation >= throttle and the full aggregate throttle — hence peak IOPS — is preserved. Only low-ring-count auxiliary devices shrink (~15*2048 = 30720, 47% of a stock 65536 budget). An explicit --device-queue-depth is honored verbatim (bypasses this path). Purely managed; no native change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Size default libaio io_setup reservation from fs.aio-max-nr via --device-aio-max-devices Extends the stock-budget reservation cap (30b71bc1) so the default libaio io_setup reservation ceiling is DERIVED from the machine's actual fs.aio-max-nr budget instead of the hardcoded 2048 cap, and exposes the provisioning target as a new knob. libaio io_setup permanently reserves io-contexts * queue-depth events from the machine-global fs.aio-max-nr budget at device creation. To guarantee a known number of Native devices always fit that budget (regardless of how a user sets --device-completion-threads / --device-throttle-limit, and including devices created off the serving factory path such as cluster auxiliary logs and AOF), ResolveLibaioReservationDepth now hard-caps each device's whole reservation (ringCount * depth) at fs.aio-max-nr / AioMaxDevices, halving the per-ring depth (staying pow2) until it fits. AioMaxDevices is a PROCESS-WIDE static (NativeStorageDevice.AioMaxDevices, default 32) because fs.aio-max-nr is a machine-global resource shared by every device in the process, so "how many devices to provision within it" is a process policy, not per-device config. Making it a static also lets devices created via the raw Devices.CreateLogDevice path honor the budget without threading the value through every factory call site. It is applied once from GarnetServerOptions.Initialize() (runs before any device is created) via the new --device-aio-max-devices option (default 32). Behavior is unchanged on both a stock 65536 budget (32 devices -> 2048 events/device, matching the previous cap) and a host that sizes fs.aio-max-nr for its workload (e.g. 4194304 / 32 = 131072/device, which never binds -> serving devices keep full depth, zero IOPS cost). io_uring is unaffected (no global budget; per-ring mmap). Purely managed; no native change. Validation: ClusterFailoverAttachReplicas 8/8 pass at fs.aio-max-nr=65536 (peak aio-nr 32768, 50% margin, no errno-11); device gate libaio 9.12M (>=8.75 floor) / uring control 8.75M (>=8.04 floor) 0 IO errors; RESP serving libaio t48 reqb1024 median 7.46M (>=7.43 floor); 89 DeviceTests pass; new DeviceAioMaxDevicesOption config test; dotnet format clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Harden native device from GPT/Gemini PR review Addresses high-value findings from the GPT-5 and Gemini reviews of this branch. All four are correctness/robustness hardening on the Native device; none change the tuned defaults, and the device + RESP perf gates reproduce the batch-free floors (libaio 8.85M / uring 8.53M device; libaio 7.35M / uring 7.29M RESP t48) with zero IO errors. io_uring submit race (Critical, both reviewers): UringFile::ScheduleOperation treated the io_uring_submit() return count (res >= 1) as proof our tail SQE reached the kernel and released sq_lock around sched_yield during retry. Both are unsafe: io_uring_submit() may partially consume under kernel backpressure (positive count < pending while our SQE is still queued), and dropping the lock lets a peer submitter sharing the ring flush our SQE, after which our own retry misreads the empty SQ as "nothing submitted", rewrites the SQE to a no-op, and frees an io_context whose IO is already in flight -> use-after-free on completion. Fix: hold sq_lock across the whole retry burst and use io_uring_sq_ready(ring) == 0 as the authoritative success signal. Drainers use a separate cq_lock, so holding sq_lock never blocks CQ draining and a transient CQ-full clears as they free space. libaio explicit sub-128 depth: the 3-arg QueueIoHandler ctor floored max_events up to kMaxEvents (128), silently over-reserving from the global fs.aio-max-nr budget whenever the managed layer deliberately passes a shallower depth to fit many coexisting single-ring devices (the --device-aio-max-devices budget math can drive depth below 128). Honor a positive max_events verbatim; only a non-positive value falls back to the default. IDevice.TryCompleteMine default: make it a default interface method delegating to TryComplete() so external IDevice implementations continue to compile and behave correctly without change; sharded devices override it. MaybeReconcileActiveShards: document the known, bounded, self-correcting transient over-subscription (a reused submitter thread seeing a stale-low divisor for at most one reconcile window) and why exact 0<->1 shard-occupancy transition tracking is deliberately avoided (cross-counter race / hot-path lock hazard). Documentation only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Website] Add Device Tuning developer guide Documents the Native storage device tuning surface: every --device-* knob with its default and meaning; the three orthogonal capacity dimensions (ring count N, ring depth D, aggregate in-flight T) and the T <= N*D invariant; the exact derived-parameter formulas (smart io-context default, queue depth, libaio fs.aio-max-nr reservation, effective throttle, per-thread sharding); the internal constants that bound them; precise definitions of headroom vs floor vs cap vs ceiling and why each exists; tuning recipes; and diagnostics. Registered in the Developer Guide sidebar. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Resp.benchmark] Add NVMe RAID-0 sample-results matrix + generator script Adds a "Sample results — 8x NVMe SSD RAID-0" section to the Resp.benchmark README with the full scenario-2 GET throughput matrix ({Libaio,Uring} x {NUMA-pinned,no-pin} x thread-count), measured on out-of-box device defaults (only --storage-tier + --device-io-backend), plus host specs, the fio 8.24M ceiling reference, and repro instructions. Peak ~7.4M ops/sec (uring, pinned, t=48) = ~90% of the fio ceiling driven end-to-end through RESP. Checks in the generator behind the table, benchmark/Resp.benchmark/scripts/nvme-raid0-matrix.sh: sweeps both backends x pin/no-pin x threads, median-of-N per cell, emits the Markdown table. Runs out-of-box defaults by default; set CT/THROTTLE/URING_IOCTX to reproduce the hand-tuned configuration. Measurement + docs only; no product code changed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 30990135225) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] Cap device NumShards at 32 and track activeShards exactly Two device in-flight-accounting refinements in NativeStorageDevice, both managed-only: 1. NumShards = Math.Min(2 * ProcessorCount, 32) (was Clamp(2*cores, 128, 1024)). A fresh device.bench + RESP shard-count ablation (NumShards 448->16 x threads 32->64, both backends) shows throughput is flat from 448 down to ~32 and only dips below ~16 -- the free-list never starves because Throttle() caps each shard's TOTAL in-flight at PerThreadLimit (<= MaxPerThreadInFlight), so a small fixed count neither starves the free-list nor re-introduces counter contention. 32 is ample headroom over the peak concurrent submitter count; the fixed per-shard tables shrink accordingly (MaxResults 8192 max). 2. Maintain activeShards exactly instead of a 200ms background reconcile. Collapse the two monotonic counters (shardSubmitted/shardCompleted) into one signed shardInFlight[] and account activeShards inline: SubmitToShard bumps it on a shard's 0->1 transition, CompleteShard drops it on 1->0, each detected atomically from the interlocked counter's own return value. activeShards is now the exact live occupied-shard count at all times -- it cannot ratchet up under .NET ThreadPool churn, so the per-thread throttle divisor never runs away. Removes MaybeReconcileActiveShards, nextReconcileTicks, ReconcileIntervalMs and the reconcile call on the hot completion path. Validation: core + GarnetServer + Device.benchmark Release build 0-warn; dotnet format clean; 89 DeviceTests pass. Device.bench (512B rand read, /raid, io-ctx32, throttle 4096) matches the ablated ns=32 numbers within noise on both backends (libaio t32 8.63M, uring t32 8.33M), 0 IO errors. RESP GET disk-serving (uring io-contexts 96, t32, successive 15s runs with 25s idle gaps) stays flat at ~6.2M across 7 runs -- the divisor-runaway decline the 200ms reconcile fixed does not return with exact accounting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Resp.benchmark] Make offline MSET serve loop allocation-free and fix flat-buffer NRE The GarnetClientSession and SERedis offline MSET runners allocated a fresh argument array every request: GetRequestArgs() returned the cached payload list, and the runner built `new string[Count + 1]` with "MSET" prepended before each Execute. That is pure hot-loop garbage on the critical serve path. Prepend the command token once, at generation time, instead of per request: - flatRequestBuffer is now List<string[]> and ProcessArgs keeps the parsed command token as element 0, so every cached entry is a complete, ready-to-send argument array ([MSET, k1, v1, ...]). - GetRequestArgs() returns that shared string[] directly. - GCS runner passes it straight to Execute (zero alloc, zero copy, no mutation). InternalExecute serializes synchronously and never retains the array, so sharing it read-only across the run's worker threads is safe. - SERedis runner iterates key/value pairs from index 1 (skips the command token). Also fix a pre-existing NullReferenceException: Run() constructed the ReqGen without flatBufferClient, leaving flatRequestBuffer null and crashing any GCS/SERedis MSET serve phase in GetRequestArgs. Mirror LightOperate and set flatBufferClient for those client types so Generate() populates the cache. Verified: build 0 warn/0 err, dotnet format clean; GCS MSET t=8 runs at ~23M ops/sec with no race or crash; SERedis MSET clean; DBSIZE and GET round-trip confirm correct 16-byte key/value pairs (command token consumed, no off-by-one). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Replace shard-counter stride indexing with a padded ShardCounter struct The per-shard in-flight counters lived in a flat long[] with each shard's counter manually spaced by ShardStride (16 longs = 128 bytes) via `shardInFlight[shard * ShardStride]` at every access site. Replace that with a typed, cache-line-padded struct element, mirroring the existing SpscRingState pattern in the same directory: [StructLayout(LayoutKind.Explicit, Size = 2 * CacheLineBytes)] struct ShardCounter { [FieldOffset(0)] public long InFlight; } shardInFlight becomes ShardCounter[NumShards]; accesses become `shardInFlight[shard].InFlight`. The 128-byte element size preserves the original spacing (each counter owns a cache-line pair, defeating false sharing and the adjacent-line prefetcher), and the generated address math is identical (base + shard*128), so this is a behavior-preserving refactor that removes the error-prone manual stride arithmetic and the ShardStride constant. Validated: - Build 0 warn/0 err; layout check confirms sizeof==128, InFlight 8-byte aligned, elements exactly 128 bytes apart, Interlocked counts exact. - Device.benchmark on 8xNVMe RAID-0 (512B random reads, t=32, throttle 4096): libaio 8.03M vs 8.11M baseline, uring 8.21M vs 8.12M baseline -- both within +/-1.1% (device.bench run-to-run noise), no regression at the ~8M ceiling. - DeviceTests: 89/89 pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Unify TryComplete(mineOnly) + cap BufferPool stripes at 32 Two device/storage changes: 1. Unify IDevice.TryComplete()/TryCompleteMine() into a single TryComplete(bool mineOnly = false) across the managed layer and the native C ABI (NativeDevice_TryComplete(device, int mineOnly); the NativeDevice_TryCompleteMine export is removed). Read paths that await their own ring pass mineOnly:true; the flush-wait keeps walk-all. linux-x64 native binaries (uring + libaio) rebuilt/redeployed; CI symbol checks updated. Other RIDs are refreshed by native-build.yml. 2. SectorAlignedBufferPool stripe count: 128 -> Math.Min(2*ProcessorCount, 32), matching the device NumShards cap. A KV.benchmark scenario-2 sweep (100% random disk reads, 8xNVMe RAID-0) shows a single pool caps at ~45% of peak (ConcurrentQueue cache-line contention) while 32 stripes matches 128 within noise across client-thread counts 32/64/96 on both libaio and io_uring; the knee is throttle-bounded, not thread-bounded. Stripe assignment switched from a pow2 bitmask to overflow-safe uint modulo. Validated: core/KV.benchmark/GarnetServer build 0/0; dotnet format clean; DeviceTests 89/89. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Add opt-in io_uring SQPOLL device knob (--device-uring-sqpoll) Adds io_uring SQPOLL (IORING_SETUP_SQPOLL) as an opt-in device knob so a kernel poll thread drains the submission queue and submissions become syscall-free. All rings share one poll thread (ring 0 spawns it; the rest attach via IORING_SETUP_ATTACH_WQ). Default off; libaio ignores it. Knob chain: native UringIoHandler ctor (sqpoll + sq_thread_idle_ms) -> native_device.h/native_device_wrapper.cc C ABI -> managed NativeStorageDevice P/Invoke + ctor -> Devices.CreateLogDevice + LocalStorageNamedDeviceFactory -> GarnetServer (--device-uring-sqpoll[-idle-ms]) and both benchmarks (KV.benchmark + Device.benchmark). file_windows.h ignores it (Linux-only). Measured DECISIVELY NEGATIVE on this 8xNVMe RAID-0: SQPOLL is ~15-23x SLOWER in every high-IOPS config (Device.bench t=32 8.27M off vs 0.52M on; KV.bench off 7.6M vs on 0.5M) because the single shared kernel poll thread serializes submission (~0.5M/s ceiling) while all submitters spin. Kept as an opt-in knob per request, but the help text, defaults.conf comment and device-tuning doc warn it is not recommended for multi-ring serving and is only useful for low-core / syscall-bound / low-concurrency workloads. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 31075026480) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] SQPOLL: one poll thread per ring + configurable CPU pinning Redesign the opt-in io_uring SQPOLL device knob so each ring gets its OWN kernel submission-poll thread instead of sharing a single one across all rings. The previous shared design (ring 0 spawns the poll thread; rings 1..N-1 attach via IORING_SETUP_ATTACH_WQ) serialized submission through one kernel thread and was a hard throughput ceiling (~15-23x slower on the 8xNVMe RAID-0). Creating every ring with plain IORING_SETUP_SQPOLL restores parallel submission. Add --device-uring-sqpoll-cpus (comma-separated CPU-id list) to optionally pin the poll threads: ring i binds to cpus[i % count] via IORING_SETUP_SQ_AFF; empty (default) leaves them unpinned so the kernel places them freely. Measured (Device.benchmark, 8xNVMe RAID-0, uring, 512B random reads, node0): per-ring SQPOLL now matches or slightly beats the default per-submit path, peaking at 8.39M ops/s (fio parity) at io-contexts=32,threads=32 vs 8.12M without SQPOLL. Leaving the poll threads unpinned (float) is the best default; static pinning is available for isolation but measured slightly worse here. Plumbs --device-uring-sqpoll-cpus through the full chain (native ctors -> C ABI -> managed P/Invoke -> factory -> Device/KV benchmarks -> host config) and updates the device-tuning doc + config test. Rebuilt the linux-x64 native .so (both variants); other RIDs regenerated by native-build.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 31082559176) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] SQPOLL: remove static CPU-pin knob (float is strictly better) The --device-uring-sqpoll-cpus knob (IORING_SETUP_SQ_AFF + sq_thread_cpu per ring) was measured strictly inferior to leaving the poll threads unpinned across every configuration on the 8xNVMe RAID-0 target: at the 32/32 peak, float held 8.39M ops/s while pinning dropped to 8.01M (and to 4.69M at 16/32). With node 0's cores mostly idle the kernel spreads the per-ring poll threads better than any static map, and pinning them onto the submitter/RESP cores costs throughput. Since there is no configuration where pinning wins here, drop the knob entirely rather than ship a foot-gun. Keeps the per-ring design (each ring created with plain IORING_SETUP_SQPOLL, no IORING_SETUP_ATTACH_WQ) which is the actual win. Removes the sqpoll_cpus parameter across the whole chain: native ParseCpuList / sqpoll_cpus_ field / SQ_AFF block (file_linux.h), the ctors (6-arg -> 5-arg in file_linux.h / file_windows.h / native_device.h), the C ABI (native_device_wrapper.cc), managed P/Invoke + ctor + field + log (NativeStorageDevice.cs), the Devices.CreateLogDevice / factory signatures, both benchmarks, and the host Options / GarnetServerOptions / defaults.conf. --device-uring-sqpoll and --device-uring-sqpoll-idle-ms are unchanged. Rebuilt both native .so (uring 2218985 -> 2206593; 18 exports each, no instrumentation). dotnet format clean; GarnetServer/benchmarks/test build 0-warn; GarnetServerConfigTests.DeviceUringSqPollOptions updated and passes; 89 DeviceTests pass; SQPOLL smoke run 0 errors. device-tuning.md updated (dropped the pin column/paragraph; float-only results table + tip). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make PR comments precise: drop dev-history and measurement narration Rewrite device/buffer-pool comments added in this branch to state durable, present-tense technical rationale instead of citing one-off profiling/sweep measurements or an older "legacy" mode: - Replace "profiled at ~13%/~26% CPU", "measured +5-7%", "measured neutral vs 65536", "sweep showed", "~45% of peak", "run-to-run noise across 32/64/96" with the underlying design facts they justify. - Reword "legacy" (default/single-ring) references to describe what the path is rather than that it is old. - Genericize the sample tiered-log path in the RESP matrix script. Comment-only changes; native .so binaries are byte-identical and untouched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 31124512411) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] Right-size buffer-pool stripes to 16 via shared sharding formula Factor the two per-thread de-contention counts — the device in-flight shard count (NativeStorageDevice.NumShards) and the sector-aligned buffer-pool free-list stripe count (SectorAlignedBufferPool.stripes) — onto a single shared sizing formula (ConcurrencySharding.Compute) so they cannot diverge, while giving each its own cap because their contention floors differ in kind: - NumShards keeps cap 32: its floor tracks the peak concurrent submitter count, so below ~32 distinct concurrent submitters collide on a shard and the per-shard in-flight counters and slot free-lists re-contend. - stripes drops to cap 16: its free-list traffic is bounded by the device in-flight throttle rather than the submitter count, so it is thread-count-insensitive and holds peak at a smaller count. RESP disk-serving GET (libaio, pinned, 100M x 128B random reads from a RAID-0 span, reqb 1024, median-of-3) is unchanged within noise at the smaller stripe count: t48 7.43M, t64 7.32M — matching the 32-stripe baseline while halving the buffer-pool free-list array. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Separate TryComplete/TryCompleteMine + address PR review comments Split the unified IDevice.TryComplete(bool mineOnly) into two distinct methods: TryComplete() walks all completion contexts/rings (the safe superset used by the allocator flush-wait), and TryCompleteMine() drains only the caller's affine context/ring (the inline submitter-thread path). TryCompleteMine() is a default interface method delegating to TryComplete(), so external IDevice implementations compile unchanged; NativeStorageDevice and the native libaio/uring handlers override both (TryCompleteBatchFor is renamed TryCompleteMineBatch). Also addresses three PR review comments: - ABI arity: split the native creator into an ABI-stable arity-9 NativeDevice_CreateWithBackend forwarder and an extended arity-11 NativeDevice_CreateWithBackendSqPoll body. The managed wrapper binds the extended symbol, so a stale native library fails fast with a clear rebuild message instead of silently reading uninitialised stack for the SQPOLL args. - Throttle docs: document that per-shard admission is approximate (aggregate in-flight can overshoot by up to ~activeShards); the native ring-full retry is the exact kernel-capacity safety backstop, not the precision of the split. - Test coverage: extend CreateNativeForTest with io-contexts/queue-depth/SQPOLL parameters and add round-trip tests for multi-ring, explicit queue depth, and io_uring SQPOLL. Includes the CI export-verify update (probe CreateWithBackendSqPoll and TryCompleteMine on Linux and Windows), knob doc/help polish, and the rebuilt linux-x64 native binaries; the other RIDs are regenerated by native-build.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Collapse native creator to a single lean export Garnet is the only consumer of libnative_device and ships the binary in-tree, rebuilt from the same commit, so the native ABI never has to stay stable across a version skew. Carry the io_uring SQPOLL parameters directly on NativeDevice_CreateWithBackend (one export) and drop the separate NativeDevice_CreateWithBackendSqPoll body plus the arity-frozen forwarder. The managed P/Invoke binds NativeDevice_CreateWithBackend and calls it directly; the create-time EntryPointNotFound guard is removed because the symbol name is unchanged and the startup probe of NumIoContexts / QueueRunFor / TryCompleteMine already rejects a stale library. CI export-verify drops CreateWithBackendSqPoll. Rebuilt the linux-x64 native binaries; other RIDs are regenerated by native-build.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 31220825798) from the native sources on 'badrishc/optimize-device-iops'. * [Docs] Align device-IO docs with final code (smart io_uring ring default + knob ranges) Correct the benchmark READMEs and configuration reference so they reflect the final shape of the Native device IO tuning surface: - Device.benchmark/KV.benchmark READMEs: io_uring no longer needs a manual --device-io-contexts to reach the ceiling. Document the smart ring-count default min(2 x cores, 64) (floored at the drainer count, decoupled from --device-completion-threads); libaio stays at rings = drainers. Update the uring example to drop the explicit --device-io-contexts and record the out-of-box default-rings result (8.45 M) alongside the under-provisioned (~2.9 M) and explicit-32 (8.00 M) rows. - Resp.benchmark README: reduce the stale scenario-2 "quick" GET table (whose uring rows predated the smart default and understated it) to the libaio rows plus a pointer to the authoritative Sample results matrix. - configuration.md: fill the two blank range cells for --device-uring-sqpoll-idle-ms ([0, 600000]) and --device-aio-max-devices ([1, 4096]) to match their [IntRangeValidation] attributes. Docs only; no code or behavior change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] NativeStorageDevice: exception-safe lazy device creation Harden EnsureNativeDeviceCreated against a partially-initialized device. The native handle (newDevice) is created but only published to the nativeDevice field at the very end, after the completion drainer threads are started. If an exception is thrown in between (e.g. Thread construction/Start under resource pressure, or the ABI-probe EntryPointNotFoundException), the old code left two problems: (a) the native handle leaked -- Dispose observes nativeDevice == IntPtr.Zero and skips NativeDevice_Destroy, so the OS file handle / io_uring rings / libaio contexts are never released; and (b) completionThreads could contain null slots for drainers that were never started, which a later Dispose would NullReferenceException on while joining (foreach ... t.Join()). Wrap the QueueRun probe, drainer spin-up, and the publishing Volatile.Write in a try/catch. On any failure, cancel + join whatever drainers were started (they spin-yield on the still-null nativeDevice field, so cancellation is observed promptly), dispose the token, reset the partial fields, NativeDevice_Destroy the handle, and rethrow. The inner EntryPointNotFoundException handler no longer destroys the handle itself (the outer catch now owns that, avoiding a double-destroy). Also make Dispose's drainer join defensive (t?.Join()). Managed-only; no native ABI change. Found independently by two code-review passes. 94 DeviceTests pass; GarnetServer builds 0-warn; dotnet format clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] NativeStorageDevice: remove redundant GARNET_DEVICE_IO_CONTEXTS env override The io_uring ring count is already fully controllable via the documented `--device-io-contexts` server option (Options.DeviceIoContexts -> numIoContexts constructor parameter), and the unset case is handled by the hardware-aware smart default. The GARNET_DEVICE_IO_CONTEXTS environment variable was a leftover tuning backdoor from the optimization phase that duplicated that control, was undocumented, and wrote to stderr on every device creation when set. Remove it; no functionality is lost. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Config] Mark device-uring-sqpoll-idle-ms as Linux-only (io_uring), matching its companion Addresses PR review: the --device-uring-sqpoll-idle-ms help text and the DeviceUringSqPollIdleMs defaults.conf comment lacked the "Linux-only, DeviceType=Native + io_uring" qualifier that its companion --device-uring-sqpoll already carries. The idle window only applies to io_uring SQPOLL, so it is just as Linux/Native/uring-specific; add the qualifier for consistency. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Group device-type-specific tuning into options objects in CreateLogDevice Addresses PR review (param list getting long; break up by device type). The Devices.CreateLogDevice parameter list had grown to 18 params as Native (libaio / io_uring) tuning knobs were added. Introduce two option objects: - NativeDeviceOptions: IoBackend, NumIoContexts, QueueDepth, UringSqPoll, UringSqPollIdleMs (Native-on-Linux backend tuning). - LocalMemoryDeviceOptions: SegmentSize, RingCapacity. CreateLogDevice now takes these instead of the eight loose device-specific params (18 -> 12). numCompletionThreads stays top-level since it is shared (Native completion drainers and LocalMemory parallelism). Behavior is unchanged: the options fields map 1:1 to the previous params with identical defaults. Updated the three call sites that passed device-specific params (LocalStorageNamedDeviceFactory.Get forwarding, and the two benchmarks' LocalMemory calls); the ~140 common call sites are unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Bundle Native tuning into NativeDeviceOptions in device factory + creator Follow-up to the CreateLogDevice refactor: apply the same "break up by device type" grouping to LocalStorageNamedDeviceFactory and LocalStorageNamedDeviceFactoryCreator, which had the same five loose Native (libaio / io_uring) tuning params (ioBackend, numIoContexts, queueDepth, uringSqPoll, uringSqPollIdleMs). Both constructors now take a single NativeDeviceOptions instead; numCompletionThreads stays top-level (shared). Behavior unchanged (1:1 field mapping, same defaults). Updated the two callers that passed the Native params (Options.GetServerOptions and GarnetServerOptions.Initialize); the ~15 other creator callers pass only common params and are unaffected. Factory.Get() now forwards the stored options object directly to CreateLogDevice. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Adopt main's scalable buffer pool; drop striped pool from this PR PR #2063 merged a scalable origin-return SectorAlignedBufferPool (with a --use-legacy-buffer-pool switch to the legacy per-level ConcurrentQueue pool). That supersedes the striped SectorAlignedBufferPool this PR had introduced, so during the rebase onto main the striping was dropped and BufferPool.cs is taken verbatim from main. This follow-up trims the leftovers: ConcurrencySharding no longer sizes a buffer-pool stripe count (StripeCount removed) — it now sizes only the device in-flight shard count (NumShardCount). Updated the NativeStorageDevice sharding doc comment that referenced the buffer pool's stripe count. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] KV.benchmark: refresh NVMe storage-bound results for the scalable buffer pool Re-measured the storage-bound sweep on the origin-return buffer pool using the command documented in the README (100M x 100B, --log-memory 16m, throttle 4096, completion threads 8, --run-threads-sweep 8,32,64, trimmed mean of 3, 8xNVMe RAID-0). Both the magnitude and the shape of the table changed: KV now peaks at ~7.8 M (~95% of the 8.24 M fio ceiling) instead of ~6.7 M, libaio scales through t=64 rather than falling off, uring peaks at t=32, and NUMA pinning lands within run-to-run noise. Updated the headline figure, the table, and the narrative to match, and noted the io-contexts setting used for the uring rows. Device.benchmark results were re-measured on the same pool and reproduce the documented figures within noise, so that README is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Address PR review: drop dead instrumentation, fix inverted docs, harden SQPOLL wakeup Removes the TSAVORITE_DEVICE_INSTRUMENT environment variable together with the counters it gated (submitCount / completeCount / peakNumPending / submitNanos) and the public GetAndResetStats() that exposed them. GetAndResetStats had no callers anywhere in the repo, so the whole block was dead weight on the submit and completion paths and the last remaining tuning backdoor in this PR; the documented control surface is the --device-* options. Corrects two comments that stated the shard-count invariant backwards. Both claimed the count "must stay at or above the peak concurrent submitter count (roughly 2 x ProcessorCount) ... so it is capped at 32", which is self-contradictory: a cap of 32 cannot enforce a floor of 2 x ProcessorCount. The cap in fact bounds MaxResults and the O(shards) TotalInFlight scan, and submitters beyond the shard count share a shard safely because the per-thread throttle already gates each shard's in-flight. Documents the real aggregate in-flight ceiling. PerThreadLimit clamps at MaxPerThreadInFlight, so device-wide in-flight is bounded by NumShards x MaxPerThreadInFlight = 4096 regardless of --device-throttle-limit. The tuning guide advised raising the throttle to 65536 for extra throughput, which the code cannot honor; that recipe is removed and the ceiling is stated in both the derivation and the sharding section. Retries the SQPOLL wakeup on every negative io_uring_submit return rather than only -EAGAIN / -EBUSY. That enter is issued only when the kernel has flagged the poll thread as parked, so any failure (for example -EINTR from a signal) can leave the SQE published with the poller still asleep; with no later submit to redeliver the wakeup the IO never completes and Dispose's drain-wait hangs. Retrying is safe because liburing recomputes the pending count from the ring. Makes Native_ExplicitIoContexts_DefaultDepth_MultiRing actually exercise multi-ring fan-out. Ring assignment is thread-affine and sticky, so issuing all 128 reads from the single NUnit thread drove exactly one of the eight rings and the test's own comment was false; reads are now submitted from eight threads. Aborts nvme-raid0-matrix.sh when the dataset load fails or comes up short. The script runs without set -e, so a failed load previously fell through to the GET sweep, which would serve in-memory misses and publish a high but meaningless NVMe throughput table. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Address PR review: exception-safe init, SQPOLL kernel guard, honour explicit uring depth Review findings from the end-to-end GPT and Gemini passes over this PR. Native: - io_uring SQPOLL now refuses at init on kernels that lack IORING_FEAT_SQPOLL_NONFIXED (pre-5.11). Such kernels only accept ring-registered descriptors under SQPOLL, so every submission of an ordinary fd would complete EBADF; failing init surfaces an actionable error (errno 95) instead of a silently broken data path. native_device.h lists the new cause in the init-failure message. - An explicit per-ring queue depth is honoured verbatim (apart from the power-of-two rounding io_uring_queue_init requires) instead of being floored at kMaxEvents. This matches QueueIoHandler and the documented --device-queue-depth semantics: a caller that asks for a shallow ring to bound per-ring memory now gets one. - -EINTR is treated as transient in both submit loops. The authoritative "consumed" signal is checked first (io_uring_sq_ready == 0 / io_submit returning 1), so the error branch is reached only when nothing was queued and a retry cannot double-submit. Managed: - The exception-safe region in EnsureNativeDeviceCreated starts at handle creation, so a throwing P/Invoke between creation and the first try block can no longer leak the native handle. The two now-redundant destroy calls are removed. - Corrected the PerThreadLimit and CompletionWorker documentation. Tests: the three native multi-ring cases submit from background threads through a shared helper, so reads actually fan out across rings and contexts rather than all landing on the single NUnit thread's affine ring. Docs: Device.benchmark option help and README sections describing ring-full handling, throttle sharding and SQPOLL now match the implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Benchmark READMEs: state the fio ceiling at both measured block sizes The Device/KV/Resp benchmark READMEs compared Garnet throughput measured with 512 B sector reads against a fio ceiling quoted only at 4 K, without saying the two block sizes are comparable. The array is IOPS-bound rather than bandwidth-bound in this range, so the same fio job yields 8.24 M IOPS at 4 K and 8.20 M IOPS at 512 B; the parity percentages are unchanged, but the READMEs now quote both figures so the comparison is explicit. Also drops the inaccurate "4 KB-class random reads" description of the RESP workload, which reads 128 B records over the array's 512 B sectors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tests] nvme-raid0-matrix: read the DBSIZE reply without blocking The load-verification guard read a fixed 32 bytes from the raw RESP socket, but the ":<n>\r\n" integer reply is shorter than that, so on a host without redis-cli the read blocked forever and the matrix never advanced past its first load. Read a single terminated line with a timeout instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tests] nvme-raid0-matrix: verify the load from the client's op count DBSIZE scans the whole hash index, so on the 100 M-key store it does not reply within the probe window and the guard read back an empty count. The loader already reports the number of ops it pushed, which is the same signal without a server round trip. Route both logs through overridable variables. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Scale the buffer pool depot stripes with the machine Disk-serving RESP GET ran ~31% below the same workload on the pre-rebase tree. A cross-commit A/B (two order-flipped rounds, three passes each, fresh 100M-key load per arm) put the regression at 6.72 M ops/s versus 4.62 M with fully disjoint ranges, and a perf call graph placed ~11.6% of server CPU in unresolved libcoreclr frames under NetworkGET_SG that were absent from the faster arm. Instrumenting the pool located it. Buffer gets split 25% owner-local / 75% shared depot, and the depot's stripes are locked stacks, so ~186 threads drove roughly 8 M Monitor.Enter per second across only 8 locks. The stripe count was fixed at 8 regardless of the hardware, so the number of threads that can enter the depot concurrently did not scale with the machine: sized for a small box it serializes a large one. It now derives from ConcurrencySharding, which already sizes the device in-flight shards this way, rounded up to a power of two so the existing mask indexing still applies and floored at 8 so small boxes keep their current width. The cap is 64, which covers the common concurrency range. A stripe sweep at 48 client threads (three order-rotated rounds of three passes, fresh 100M-key load per run) gives 4.51 / 5.15 / 5.73 / 6.85 / 6.82 M ops/s at 8 / 16 / 32 / 64 / 128 stripes, so throughput is flat from 64 onward there. Because a server drives the depot from roughly one thread per connection, the knee does move out at much higher connection counts: at 128 client threads (~266 server threads) 64 stripes gives 5.85 M against 6.70 M at 256. That degradation is accepted in exchange for the smaller stripe array and shorter miss scan; workloads that sustain far more concurrent threads than the cap trade some throughput for those bounds. Widening does not strand buffers, because a depot miss already scans every stripe of the size class rather than only the caller's; the extra stripes cost about 113 KB per pool and nothing per operation. No other pool constant changes, so the byte budget still bounds retained memory exactly as before. Raising LocalCap alongside this was measured and dropped: at the wider stripe count, 128 / 256 / 512 land at 6.86 / 6.96 / 7.02 M ops/s over three order-rotated rounds of three passes, so the larger caps buy 1.4-2.3% while raising the per-thread hoard bound, which nothing steals from until the thread dies. The stripe count alone restores throughput. The full NVMe matrix on device defaults peaks at 7.36 M ops/s (uring, pinned, t=48), reproducing the published table within 4% on every pinned cell, with the pool's fresh-allocation rate falling to zero. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Trim device docs and comments to first-principles statements Remove narration of prior behaviour, rejected alternatives, and measurement history from the device tuning page, the benchmark README, and the device and buffer pool comments, and cut the restatement that followed the floor / cap / ceiling / headroom definitions. Each remaining statement describes what the code does now and the constraint that shapes it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Harden the native device against review-surfaced failure modes End-to-end review of the device changes (agent plus two independent model reviews) surfaced five real defects. Each is fixed here; measurements below confirm no throughput cost. Use-after-free on the submit path. ReadAsync/WriteAsync bump their shard's in-flight count before the P/Invoke, but that bump is dropped by the completion callback running on a drainer thread. A fast completion can therefore drive the count to zero while the submitting thread is still inside native code -- still to run ~EpochGuard, which touches the device's epoch. Dispose can then observe zero in-flight, join the drainers and destroy the device under the returning submitter. Both entry points now take a second, independently balanced lease around the native call, the same guard the non-IO entry points already get from TryLease. io_uring drainer raced SQ submission on kernels before 5.11. Without IORING_FEAT_EXT_ARG, liburing emulates io_uring_wait_cqe_timeout by taking an SQE, writing a timeout request with a reserved user_data and flushing the SQ -- from the completion side, without sq_lock, while submitters mutate the same SQ. The reserved user_data is also non-null, so the batch dispatcher would have treated it as a caller context. The feature bit is now sampled at init and the drainer polls the completion queue instead when it is absent. SQPOLL wakeup could be dropped. Once io_uring_submit publishes an SQE to an SQPOLL ring the kernel owns it, so a failed enter means only that the wakeup was not delivered to a parked poll thread; with no later submit on that ring the IO never completes. The retry now backs off through sched_yield into bounded 1 ms sleeps rather than giving up after a few yields. Thread-start failure leaked the native device. The drainer slot was published before Start(), so a failure under resource pressure left an unstarted thread in the array; the cleanup path's Join() then threw ThreadStateException, escaped, and skipped the destroy. The slot is now published after the thread is running and the destroy runs in an unconditional finally. A transient startup probe error published a device with no drainers. NativeDevice_QueueRun doubles as the capability probe: Windows IOCP returns a permanent negative, but a Linux backend can return a transient negative when the probing thread is interrupted by a signal, which the runtime does routinely. It is now retried before concluding the backend has no drainable queue. Also: clamp RoundUpPow2 at IORING_MAX_ENTRIES so an out-of-range depth through the C ABI cannot overflow; guard the completion callback's own logger call so a throwing host logger cannot defeat the drainer firewall; and warn when a libaio reservation cannot be brought within its per-device share of fs.aio-max-nr, since depth cannot fall below one event per ring. The AioMaxDevices help text and docs no longer claim an unconditional guarantee. Device.benchmark, libaio, 512 B random reads on 8x NVMe RAID-0, 32 threads, 6 interleaved samples per arm: 8.469 M ops/s with the fixes against 8.527 M before them, with fully overlapping ranges. The two extra interlocked operations land on a shard line the submitter already owns. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Align the libaio budget cap description with its actual guarantee The `--device-aio-max-devices` section claimed the per-device reservation cap guarantees at least that many devices can be created "regardless of the other knobs". The cap cannot go below one event per ring, so a device configured with more rings than its per-device share still exceeds it, and the budget is the machine total rather than what remains after other processes. The derivation section and the option help text already state both limits; this makes the knob section agree and links to the derivation. Also record the second reservation warning in the many-devices recipe: one fires when `N x D` exceeds `fs.aio-max-nr`, the other when a device cannot be brought within its per-device share. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Correct the libaio reservation floor description The floor glossary entry claimed the reservation depth is never sized below 128, but the per-device AIO-budget clamp runs after the floor and halves past it when the whole-device reservation does not fit. On a stock 65536 budget, 32 io-contexts resolve to a depth of 64. State the precedence in both the doc and the constant's XML doc: the budget ceiling overrides the floor, because exceeding the budget fails device creation while a shallow ring only costs throughput. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Record that the AIO-budget ceiling can reduce the effective throttle The libaio reservation notes claimed multi-ring serving devices keep the full aggregate throttle at no IOPS cost. That holds for the throttle-share math, but the per-device fs.aio-max-nr ceiling runs last and caps effectiveThrottleLimit at ringCount * depth. On a stock 65536 budget that bound is 2048, halving the default 4096 throttle at every ring count. Qualify the claim in the doc and in the three matching code comments, and give the operator the sizing rule: fs.aio-max-nr / --device-aio-max-devices >= throttle. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] State the slot free-list headroom in terms of the shard Throttle() gates a shard's whole in-flight against a limit clamped to MaxPerThreadInFlight, so a shard holds at most that many slots however many submitter threads share it. The SlotsPerShard summary attributed the bound to a single submitter instead, which reads as if the free-list could be drained by several threads sharing one shard. Match the framing already used by Throttle() and RentSlot(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Scope the reservation share-math claim to the share clamps The consequences list stated the no-IOPS-cost property unconditionally and retracted it three bullets later, so a reader on a stock budget -- where the ceiling always binds -- takes away the wrong default. Attach the property to the share clamps that provide it and point forward to the ceiling that runs after them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 31969160992) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] Drop two unreachable retry paths from the native device An audit of the review-driven hardening found two of its retry budgets guard failure modes that cannot occur. The startup QueueRun probe retried on the premise that a Linux backend can answer with a transient negative when a signal interrupts the probing thread. A zero timeout never blocks, so neither backend can: libaio passes a zero io_getevents timeout and io_uring reads the completion queue in user space without a syscall. A 3,000,000-iteration harness that hammered io_getevents with a zero timeout while another thread signalled the caller through a handler installed without SA_RESTART observed zero negatives and zero EINTR. The probe is back to a single call. The io_uring SQPOLL wakeup path gained a second backoff stage of 1000 one millisecond sleeps. An enter carrying only IORING_ENTER_SQ_WAKEUP never waits, so it cannot return EINTR, which io_uring_enter(2) documents only for IORING_ENTER_GETEVENTS; every other error on that path is permanent. Sleeping cannot turn such a failure into a success, and sq_lock and the epoch are held throughout, so the stage only held the ring for a second before reaching the same outcome. The bounded yield budget is restored. Both comments now state the mechanism that actually applies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 31975517483) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] Bound NativeStorageDevice.Dispose's in-flight drain Dispose waited for in-flight IOs with an unbounded `while (TotalInFlight() != 0) Thread.Yield();`. In-flight only returns to zero if the kernel completes every accepted IO, so a completion that is never delivered spins there forever: an unkillable teardown that pins a core and reports nothing. Lost completions are reachable from several directions — a stalled device or driver, a dropped CQE, or an io_uring ring whose SQPOLL thread has died, after which the ring accepts submissions whose completions never arrive. The drain now runs against a deadline. It sits orders of magnitude above any legitimate drain: outstanding IOs are already queued in the kernel, and every native call a lease is held across is individually bounded (the submit paths unwind after a fixed yield budget, QueueRunFor takes a timeout), so reaching it means completions are lost rather than slow. On expiry the count is logged and teardown proceeds, which is safe because the drainers are cancelled and joined before the handle is freed — no user callback can run during teardown — and NativeDevice_Destroy cancels or waits for whatever the kernel still owns. SpinWait replaces the bare Thread.Yield() so the normal microsecond drain stays spin-fast while a drain that runs to the deadline does not pin a core. Verified by injecting a phantom in-flight count with no matching completion: before, Dispose did not return within 150s; after, it returns at the deadline. file_linux.cc: comment only. The SQPOLL submit path noted that a later submit redelivers the wakeup, which holds only while the poll thread is parked. Once it is gone nothing redelivers. Failing those IOs individually would not restore correctness — everything already in flight on that ring is lost with it, and the kernel holds the only reference to their contexts (their user_data), so there is nothing to enumerate. The comment now states that and points at the drain bound. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Resp.benchmark: republish the NVMe matrix from verified-load runs The published RESP matrix was measured by a version of the generator that ran the GET sweep unconditionally after the MSET load, without checking that the load had written the dataset. A key that was never written is answered from memory with no device IO, and a miss is roughly 30x cheaper than a disk read, so a partial load inflates the reported figure: on this array an unloaded store reports over 150 M ops/sec against a true storage-bound 6.3 M, and a ~35% shortfall alone accounts for the previously published 7.36 M. Re-measured every cell with the load verification in place (the loader reports 99,876,864 of 100 M ops). The pinned peak is 6.96 M (uring, t=48) rather than 7.41 M, both backends now peak at t=48 instead of libaio climbing through t=64, and the no-pin rows drop further, which widens the NUMA-pinning gap. The new figures are also physically coherent with the neighbouring layers, which the old ones were not: raw device ~8.4 M > KV ~7.8 M > RESP ~7.0 M. Also tighten the generator's own short-load guard from 90% to 99% of DBSIZE. The 90% floor still admitted an ~11% overstatement; at 99% the reported figure is within ~1% of the fully-loaded value. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] KV.benchmark: correct the NVMe storage-bound results The published storage-bound table was measured against a build whose buffer pool differs from the one this branch ships, so it does not reproduce. Every row is replaced with a fresh trimmed-mean-of-3 measurement on the current tree, and the peak claim drops from ~7.8 M (95% of fio) to ~6.3 M (77%). The uring rows now use the smart ring default instead of an explicit --device-io-contexts 32, which under-provisions at t=64; the default is faster in every cell and makes the table match the documented command. Also corrects two claims the new data contradicts: NUMA pinning is within noise for the disk-bound scenario (it gates the RAM-served scenarios), and the three benchmarks use different datasets, so their numbers do not form an ordering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Size the buffer pool's owner-local chain to a thread's IO pipeline SectorAlignedBufferPool keeps a per-(thread, class) chain of returned buffers and spills to a lock-guarded depot once the chain reaches LocalCap. A caller that rents many buffers before returning any therefore hits the depot for everything past the first LocalCap rents, however small its actual working set. KV.benchmark issues --batch-size (default 1024) reads per iteration before draining, so its per-thread burst is 1024 buffers against a LocalCap of 128: 7 of every 8 rents and returns went through the depot. Instrumenting the pool during a disk-bound run measured 147.4M depot pops against 49.3M local hits (74.8% of gets) and 147.6M depot spills against 196.9M local pushes, with zero cross-thread and zero large-class traffic - about 1.09B Monitor operations, 6.6 per KV operation. User CPU per operation was 2.92x the raw device path (3.011us vs 1.033us) while kernel CPU per operation was lower, so the cost was managed-side, not IO. Raising LocalCap to 1024 admits the whole burst. Both consumers improve: KV.benchmark, 100M x 100B on 8xNVMe RAID-0, 100% random reads from disk, trimmed means of 3 (ops/sec): backend pin t=8 t=32 t=64 libaio node-0 2,366,537 6,861,507 7,707,696 libaio none 2,391,726 6,903,713 7,650,788 uring node-0 2,296,988 7,621,662 7,226,697 uring none 2,266,429 7,722,915 7,368,255 Peak 7.72M against 6.34M before, +21.8%, and 94% of the array's 8.24M fio ceiling. Sweeping LocalCap alone at t=32 traces the burst exactly: 128 -> 5.738M, 512 -> 6.786M, 1024 -> 6.942M, 2048 -> 6.948M, 4096 -> 6.910M, i.e. throughput follows min(LocalCap / 1024, 1) and flattens once the cap covers the batch. RESP GET, 3 rounds with the arms rotated each round, medians: LocalCap t=48 t=64 128 6.725M 6.423M 1024 7.398M 7.216M 2048 7.229M 7.182M The 128 and 1024 ranges are disjoint at both thread counts. 2048 is below 1024, so 1024 is the value both consumers want. Peak RSS falls: 9,135,776 kB at 128 against 8,930,012 kB at 1024. At the lower cap the depot overflows and the pool drops and re-allocates buffers continuously (dropped-because-full tracked fresh allocations one for one); covering the burst collapses the fresh-allocation rate to near zero. The chain is intrusive - it links through the buffer's own next pointer - so a larger cap costs no structural memory, and LocalByteCap (32 MB per thread and class) remains the bound that stops one thread parking the budget. This reverses the 2026-08-15 measurement that kept main's 128. That study swept 128/256/512 and read +2.32% at 512, inside the "under 5%, keep main's setting" band. It was under-ranged: the RESP scatter-gather path rents about 2000 buffers per network batch, so 512 covers only a quarter of the burst. The arms that cross the burst threshold are worth 10-12%, well outside the band, so the same rule now selects the change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Bound the buffer pool's owner-local retention in bytes The pool budgets bytes, but the owner-local chain was bounded by a count. A count cannot bound a byte budget: one size class's buffer is up to 512x another's, so the same count means 512x the bytes depending on which class a thread happens to use. The count a thread actually needs is its IO pipeline depth, which is a property of the caller, not of the pool. The mismatch had a reachable consequence. `permitBytes` is reserved for the life of a parked buffer, so per-(thread, class) retention was min(LocalCap x classBytes, LocalByteCap). Summed over the 16 small classes at LocalCap 1024 that is 302 MB against a 256 MB small sub-budget, so a single thread could exhaust it. Once `TryReserve` fails, every thread runs non-cacheable - every Get allocates and every Return frees - and nothing trims the thread holding the bytes. Replace both caps with a per-thread byte ceiling: the small sub-budget divided by `ConcurrencySharding.ExpectedConcurrentThreads` (min(2 x cores, 64)), floored at 1 MB. A thread's classes share that ceiling work-conservingly, so a single-class thread gets all of it. At the ceiling, `TryMakeRoom` reclaims from the class furthest above its equal share (max-min fair) and refuses the request only if the requester is already at or above its own share, which also makes self-eviction impossible. Victims are spilled to the depot, not dropped, so they stay allocated, budgeted and reusable. `ThreadShard.activeClasses` is maintained incrementally so the common single-class case tests the ceiling in O(1) - this matters because a thread at steady state sits at the ceiling and enters that path on every return. Worst-case per-thread retention drops from 302 MB to 4 MB here (75x). The smaller ceiling does not increase churn, because the real bound is the caller's IO pipeline depth and both workloads sit under the slice (KV ~1.15 MB, RESP ~2.3 MB derived from their burst sizes and class mix). Measured on 8xNVMe RAID-0, pinned, median-of-3, against the count-cap arm: RESP libaio t=48/64 7.094/7.095 (was 6.923/7.167), uring 7.389/7.030 (was 7.291/7.013); KV libaio t=64 7.76 M; peak RSS 8,927,844 kB (was 8,930,012 kB). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Resp.benchmark: refresh the NVMe storage-bound matrix The published figures were measured before the buffer pool's owner-local retention was sized to a thread's IO pipeline, so they understated libaio by up to 7% (pinned t=48: 6.51 -> 6.97 M) and reported the wrong peak thread count for libaio. Re-measured the full 16-cell out-of-box matrix with the generator script (median of 3, load verified at 99,876,864 of 100 M keys) and republished every cell: backend NUMA t=8 t=32 t=48 t=64 Libaio srv-0/cli-1 1.82 5.70 6.97 7.06 Libaio no pin 1.38 5.09 5.72 5.57 Uring srv-0/cli-1 1.82 6.13 7.32 6.89 Uring no pin 1.48 4.60 5.55 6.29 Derived claims corrected along with the numbers: - Peak is 7.32 M (uring, pinned, t=48), ~89% of the array's fio ceiling, not 7.0 M / ~84%. - The backends no longer peak at the same thread count: uring peaks at t=48 and eases off at t=64, while libaio is still climbing at t=64. The guidance is now to sweep the t=48-64 band. - The backend-parity bullet quantifies the spread (uring leads 5-7% at t=32-48, libaio by 2% at t=64) instead of claiming "within a few percent", which the t=32 cell no longer supports. - The tuned-vs-default claim drops its numeric bound, which was not re-measured on this binary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Resp.benchmark: correct the backend-gap range The published matrix shows uring ahead of libaio by 7.5% at t=32 and 5.0% at t=48, and libaio ahead by 2.5% at t=64. State 5-8% and ~2% so the bullet matches the table above it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Buffer pool: describe the per-thread byte cap The website design doc still described LocalCap (128 buffers) and LocalByteCap (32 MB) bounding a single (thread, size-class) local stack. Neither constant exists: local retention is now bounded in bytes per thread, shared across that thread's size classes, at smallBudget / ExpectedConcurrentThreads floored at MinThreadLocalBytes (4 MB at the 1 GiB default), with max-min fair admission across classes via TryMakeRoom. Also correct the summary's '8-way' depot striping, which contradicted section 6 (8-64 stripes, sized from the processor count), and note that a spill relocates a buffer to the depot with its permit intact rather than making it uncacheable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Buffer pool: state the design rather than defend it The depot section justified its lock against ConcurrentStack point by point, and the large-class section closed with before/after reuse, allocation and RSS figures from an earlier iteration. Both read as defenses of past decisions rather than a description of the design. Keep every substantive fact - atomic close, exact capacity bound, allocation-free push, and why large classes have no per-thread locality to exploit - and state them as properties of the final design. Align the equivalent source comment in ReturnOriginReturn. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Report a permanent io_uring SQPOLL wakeup failure On the SQPOLL submit path the retry loop re-enters io_uring_enter until the wakeup is delivered or the yield budget is exhausted. On exhaustion the entry is already published to the kernel, so the operation is reported submitted and the loop's last errno was discarded, leaving a ring that can accept IOs whose completions never arrive with no signal to the operator. Emit the errno and its consequence once per device, claimed through an atomic flag so a ring that fails for every submission reports a single line rather than one per IO. The report does not change the submitted outcome: the entry is kernel-owned and the kernel holds the only reference to the affected contexts via their user_data, so there is nothing to enumerate and rewriting the SQE would race the poll thread. NativeStorageDevice.Dispose already bounds its drain, so a ring in this state cannot hang teardown. Rebuilt the linux-x64 uring binary. The libaio variant is unchanged: this code is inside the FASTER_URING guard, which that build does not define. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 32199655122) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] Gate native handle destruction on native-call leases; fix two benchmark option bugs Addresses the fresh Copilot review on #2018. NativeStorageDevice.Dispose(): the shard in-flight counter was overloaded for two things with different termination properties — IOs awaiting a completion (which can be lost forever, hence the bounded drain) and leases held by threads executing inside native code. TryComplete/TryCompleteMine hold a lease across a native call that dispatches user callbacks inline, so a lease is bounded only by user code, not by the "every native call is bounded" claim the drain comment made. A lost completion could therefore trip the deadline while a native frame was still running, and the subsequent NativeDevice_Destroy would free its rings and locks underneath it. Leases are now counted separately (a second field on the existing padded shard counter, so no extra cache miss) and handle destruction waits on that counter after the in-flight drain. Leases are bumped after in-flight and dropped before it, so leases <= in-flight always holds and the normal path costs one extra read. No new lease can be acquired once disposedFlag is published, so the wait only covers calls already in native code; if it does expire the handle is leaked rather than freed, which is bounded and diagnosable where a use-after-free is not. Device.benchmark: --device-throttle-limit had no effect on LocalMemory. LocalMemoryDevice does not override StorageDeviceBase.Throttle() (which returns false), so its in-flight bound is the per-submitter SPSC ring, and RingCapacity was left at 0 => 1024. Map the throttle onto the ring capacity as KV.benchmark already does, and print the resolved value. Resp.benchmark: --load-threads 0 reached DbSize % loadDbThreads and crashed with DivideByZeroException; reject values below 1 up front. file_linux.cc is comment-only: state that io_uring_sq_ready() is sqe_tail-khead, measured against the kernel head rather than ktail, so submit's flush cannot drive it to zero and it remains an exact "kernel consumed our SQE" test under short submits and failed enters. No behavior change, so the prebuilt binaries are unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Document the t_shards/slotIndex/registry relation in BufferPool Add a section comment explaining how the two indexes over the (pool x thread) shard matrix relate: t_shards is the thread-side index (ThreadStatic, strong refs, hot path), the registry is the pool-side index (weak refs, cold path), and slotIndex is the recyclable key joining them -- which is what makes the identity check on every read necessary. Rename registry -> poolShardRegistry (and its lock, threshold, and compaction helper) so the pool-side index is distinguishable from the thread-side one at each use site. * [Tsavorite] Fix wording in slotIndex comment for consistency --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> | 18 天前 | |
[Storage] Optimize IOPS for RAID-0 NVMe disks (#2018) * Optimize NativeStorageDevice random-read IOPS (4.14M -> 6.94M on 8x NVMe RAID-0) WIP: raise Garnet SSD random-read serving throughput toward the fio ceiling (8.24M IOPS). Verified on KV.benchmark scenario 2 (100M x 100B, log-mem 16m, 100% random 4K reads from disk, native libaio), standalone runs, node0-pinned. Fixes in this branch: - FIX#1 (NativeStorageDevice.cs): shard per-submitter in-flight tracking; removed the global numPending counter + freeResults queue that cache-line ping-ponged across all submit/complete threads. Positive thread scaling. - FIX#2 (Device.benchmark/BenchWorker.cs): drop the per-op device.TryComplete() from the submit hot path (serialized all submitters on ctx0's kernel ring_lock); dedicated drainers do all reaping. - FIX#3 (cc/device/thread.h, thread_manual.cc): cache-line-pad Thread::id_used_[] (per-IO EpochGuard acquire/release CAS was false-sharing). Native .so rebuilt. - FIX#4 (NativeStorageDevice.cs): per-shard free-list for completion slots. The prior counter-ring slot reuse was unsafe under OUT-OF-ORDER device completion (a slow IO's slot could be overwritten by newer submits wrapping the ring), corrupting the AsyncIOContext and crashing KV. Slots now return to a free-list only after their own IO completes. - FIX#5 (Utilities/BufferPool.cs): stripe SectorAlignedBufferPool's per-level free-list across 128 sub-queues, thread-affine. The single ConcurrentQueue per size-level was 52.6% of all CPU (TryDequeue+TryEnqueue) under the pending-read workload. 4.14M -> 6.30M @32thr. General win for all Garnet disk reads. - FIX#6 (NativeStorageDevice.cs, Devices.cs, benchmarks): decouple num_io_contexts from num drainer threads via new numIoContexts option (--device-io-contexts / --io-contexts). Default 0 == legacy 1:1 (byte-for-byte unchanged). Multi-ring drainers range-POLL their rings (never block on one ring, which would starve the siblings). Helps low/medium thread counts; neutral at peak. Peak verified: 6.94M ops/s @48 threads (84% of fio). All 89 DeviceTests pass. TODO (not in this commit): batched io_submit; io_uring IOPOLL + registered buffers/files; patchelf .so libaio.so.1t64 -> libaio.so.1 + libaio-only variant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add opt-in device tuning levers: batched libaio submit + affine inline drain Two additional, opt-in (default = legacy behavior) levers explored while closing the KV random-read gap on 8x NVMe RAID-0. Both were measured neutral at the core-saturated peak (~6.9M) but positive at lower/medium thread counts, and are kept behind env flags with zero-regression defaults. FIX#7 batched libaio submit (GARNET_SUBMIT_BATCH, default 1 = immediate submit): - Native (file_linux.cc/.h, native_device.h, native_device_wrapper.cc): per-submitter thread_local accumulation of prepared READ iocbs, flushed via io_submit(ctx, N) at a threshold; handles partial submit / EAGAIN / per-iocb permanent error. New NativeDevice_FlushSubmits C ABI (uring backend = no-op). - Managed (NativeStorageDevice.cs, IDevice.cs, StorageDeviceBase.cs): FlushSubmits() + P/Invoke; TryComplete() flushes the calling thread's batch first (throttle-spin safety net). KV benchmark (KvBenchmark.Worker.cs) flushes the sub-threshold tail per read batch. Affine inline drain (GARNET_INLINE_DRAIN_AFFINE, default off): - New IDevice.TryCompleteMine() (StorageDeviceBase falls back to TryComplete; native NativeStorageDevice drains only the caller's affine context/ring). The inline submitter-thread completion path (TsavoriteThread.cs, AllocatorBase.cs) uses it when the flag is set, cutting per-context io_getevents syscalls at lower thread counts. Rebuilt both prebuilt Linux natives from current source (USE_URING=ON -> libnative_device.so, USE_URING=OFF -> libnative_device_libaio.so) so both export the new NativeDevice_FlushSubmits / NativeDevice_TryCompleteMine entrypoints; TryComplete calls FlushSubmits unconditionally, so the libaio-only fallback must have them too. Tsavorite build clean (0 warnings); 89 DeviceTests pass; default path is byte-for-byte legacy behavior (both env flags unset). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Resp.benchmark data races that crashed multi-threaded loads Two concurrency bugs made the offline benchmark crash (NullReferenceException in ReqGen.GetRequestArgs) and produce malformed requests under load: 1. Shared request list mutated in place. GetRequestArgs() returns a reference to a cached List<string> owned by ReqGen. GarnetClientSessionOperateThreadRunner did reqArgs.Insert(0, "MSET") on it every iteration. That both (a) raced across threads (concurrent List.Insert corrupts the backing array) and (b) permanently prepended "MSET" to the shared list, growing it unboundedly even single-threaded. Fix: build a fresh args array with the command prepended; never mutate the cache. 2. Non-thread-safe System.Random for serve-offset selection. GetRequest() and GetRequestArgs() called a shared System.Random ('r') concurrently from all worker threads, which can return out-of-range indices. Fix: use Random.Shared (thread-safe) for the concurrent serve-offset draw. 'r' is retained for the single-threaded generation phase where its deterministic seed matters. Also add --load-threads (default 8) to parallelize the initial data-load phase. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make affine inline device drain the default (fix RESP read-throughput degradation) The inline submitter-thread completion drain (Tsavorite CompletePending / AsyncGetFromDisk throttle-wait) is the primary reaper for disk-bound reads. The legacy inline drain, IDevice.TryComplete(), reaps only a single fixed io_context (context 0), so every inline-draining thread serialized on that one context's kernel aio mutex. On the RESP GarnetServer, disk-read completions are processed by the .NET thread pool, which grows under disk latency. With many workers all draining context 0, ~46% of server CPU was spent spinning on the aio mutex (osq_lock), and 100% random-read throughput spiralled downward run-over-run (6.75M -> 2.4M ops/s on an 8x NVMe RAID-0) as the pool grew. The fixed-context drain also only covered context 0's 1/N share of completions inline. IDevice.TryCompleteMine() reaps the calling thread's own affine context (the one its submits land on), spreading the inline drain across all contexts. This removes the mutex storm (osq_lock 46% -> 7%) and holds throughput stable at ~7.2M ops/s across many runs with no degradation. It was previously measured neutral at the uncontended saturated peak, so making it the default is a strict improvement. Devices that do not shard completions fall back to TryComplete() automatically, so this is safe for all device types. Flip Constants.InlineDrainAffine to default on; set GARNET_INLINE_DRAIN_AFFINE=0 to restore the legacy fixed-context-0 drain. 89 DeviceTests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add opt-in io_uring batched submit (device-level +14%, uring reaches libaio parity) Defer io_uring_submit to coalesce many read SQEs into one submit syscall, mirroring the existing libaio batch. Opt-in via GARNET_SUBMIT_BATCH (default 1 = submit per-op = byte-for-byte legacy, zero regression). Only a thread that solely owns its ring defers (per-ring CAS ownership); ring-sharing threads (submitters > rings) fall back to per-op submit. Writes never batched. Every deferred SQE carries its io_context as user_data, so exactly one completion is dispatched regardless of which thread flushes. The managed TryComplete/TryCompleteMine (and the AsyncGetFromDisk throttle-spin) already call NativeDevice_FlushSubmits before draining, upholding the flush-before-wait invariant (no throttle deadlock) with no managed changes. get_sqe==null flushes the pending batch before retry/unwind; a real UringIoHandler::FlushSubmits with -EBUSY drain-assist (TryCompleteFor) replaces the previous no-op stub. Device.benchmark (512B random reads, t=32, io-contexts=32, no pin): uring 7.85M -> 8.94M at batch=32 (+14%), a new uring high reaching ~parity with libaio-batched (9.23M); integrity verified (71.5M ok == submitted, 0 err). Confirms per-op submit was uring's disadvantage vs libaio's batched io_submit. RESP serving is unchanged (closed-loop, managed-CPU-bound; batching neutral). BenchWorker.cs: flush the deferred tail batch via TryComplete in the shutdown drain so a sub-threshold tail never strands the exit. Native .so rebuilt (uring + libaio variants; libaio path is functionally unchanged). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add GARNET_DEVICE_IO_CONTEXTS override to decouple io_uring rings from drainers Give each submitter thread its own ring (rings >= completion threads) to avoid the shared-ring non-owner submit path, independently of the drainer count. The value is clamped up to --device-completion-threads so every drainer owns at least one ring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update device/KV/RESP benchmark READMEs with current run instructions Document the device tuning flags (--device-io-contexts, --device-completion-threads, --device-throttle-limit, --device-io-backend) and the record/page/segment settings used to reproduce the current SSD random-read numbers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Batch-reap io_uring completions in UringIoHandler::TryCompleteMine The inline submitter-thread completion path (Tsavorite CompletePending / AsyncGetFromDisk throttle-wait) drains the caller's own affine ring via TryCompleteMine, which reaped exactly one CQE per call (TryCompleteFor -> io_uring_peek_cqe). The dedicated drainer's QueueRunFor already batch-reaps up to kCqeBatch CQEs per cq_lock section, and libaio's QueueIoHandler::TryCompleteMine already batches via io_getevents; the uring inline path did not, so the network thread paid one cq_lock + one dispatch loop iteration per completion on the hot critical path. Add UringIoHandler::TryCompleteMineBatch: a non-blocking single pass that reaps up to kCqeBatch (64) completions in one cq_lock section (io_uring_peek_batch_cqe -> snapshot -> io_uring_cq_advance -> release -> dispatch outside the lock), mirroring QueueRunFor's phase-2. TryCompleteMine now calls it. One managed NativeDevice_TryCompleteMine P/Invoke therefore delivers a batch of completions, which the same thread drains inline. On disk-served RESP GET (100M x 128B, uring, 96 rings, 2 drainers) this raises throughput about 20% at the t=48 peak (5.94M -> ~7.1M ops/s) and removes the prior drainer-count sensitivity (the network threads now self-drain in batches, so a single background drainer no longer collapses under throttle-spin). The libaio backend is unaffected (its TryCompleteMine already batches); the UringIoHandler code is compiled only under FASTER_URING, so the libaio-only prebuilt is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add GARNET_URING_BATCH_REAP opt-out gate for the TryCompleteMine batch-reap Make the io_uring completion batch-reap (TryCompleteMineBatch) an opt-out knob so its throughput impact can be measured on the same binary without a revert-build. Default ON (unchanged behavior); GARNET_URING_BATCH_REAP=0 falls back to the legacy single-CQE reap (TryCompleteFor). Read once via a function-local static. Consistent with the other env-gated device levers (GARNET_DEVICE_IO_CONTEXTS, GARNET_SUBMIT_BATCH). Same-build ablation shows batch-reap is neutral at the saturated peak (a harmless CPU efficiency, not a peak-throughput lever), so this gate documents that finding and keeps it toggleable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix NativeStorageDevice throttle-divisor runaway that slowly starves device queue depth The per-submitter-thread in-flight sharding split the global ThrottleLimit into a per-thread budget via PerThreadLimit() = ThrottleLimit / activeShards. activeShards was Interlocked.Increment-ed the first time each thread submitted (AssignShard) but never decremented. Under the .NET ThreadPool, submitter threads are transient: they retire when idle and fresh ones are injected on the next burst. Each fresh thread bumped activeShards, so over a long-lived device the divisor ratcheted up without bound even though the number of concurrently active submitters stayed roughly constant. As a result the per-thread in-flight budget collapsed over the process's lifetime (e.g. 4096/35 -> 4096/245), so the aggregate device queue depth starved from ~ThrottleLimit down to a small fraction of it, and disk-serving read throughput declined progressively (observed ~55% over successive RESP GET runs), recoverable only by restarting the server. In-memory workloads were unaffected because they never exercise the pending-read throttle path. Fix: keep AssignShard's immediate increment (new threads instantly get a fair share) but periodically reconcile activeShards DOWN to actual shard occupancy (shards with in-flight > 0, ~= concurrently active submitters) via MaybeReconcileActiveShards(), time-gated to at most once per 200 ms by a cheap non-atomic tick check on the completion path. Births are counted immediately; deaths are reclaimed lazily from live occupancy. This only resizes the throttle divisor (a perf knob) and does not touch slot allocation, the submitted/completed balance, or completion routing, so it cannot affect correctness. Validated: RESP disk GET is now flat across successive idle-gap-separated runs (was a monotonic ~55% collapse); 89 DeviceTests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add opt-in LightEpoch-style io_uring ring affinity (GARNET_RING_LE_AFFINITY) The committed uring model owns a ring per submitter thread via a sticky `ring_owner_[idx]` that is released only at teardown. On .NET's oversubscribed ThreadPool this orphans rings: a retired thread leaves its tid in the owner slot forever, so that ring can never batch-own again and degrades to per-op submit ("dead-tid poisoning"). This adds an opt-in ownership model that maps LightEpoch's thread-affinity pattern onto ring ownership, env-gated by GARNET_RING_LE_AFFINITY (default off = byte-identical legacy path): - thread-local preferred ring (== LightEpoch startOffset1), re-acquired warm each network batch; - probe-and-replace on collision (== TryAcquireEntry circling the table): CAS-claim a free ring and adopt it as the new preferred; - release at the network-batch boundary (== Release-on-Suspend): the reliable "suspend" point after which the thread will not submit again until its next batch, so a churned-away thread's ring is reclaimed instead of orphaned. The batch-boundary release is gated on actual disk-read activity: a `[ThreadStatic]` flag is set in NativeStorageDevice.ReadAsync (the single choke point every disk read funnels through; pure in-memory hits never reach it) and consumed by EndBatchReleaseRing() in RespServerSession's batch finally. A batch that touched no disk pays one thread-static check and no P/Invoke, so the feature is free on in-memory and mixed workloads. Native: file_linux.{h,cc} add le_affinity_enabled() (cached), pick_ring_index_le() (warm fast-path / CAS-claim / probe-adopt), release_my_ring(), and a unified uring_thread_id() so pick and submit agree on the owner id; libaio's QueueIoHandler gets a no-op release_my_ring(). native_device.{h} + native_device_wrapper.cc export NativeDevice_ReleaseRing. AllocatorBase / LogAccessor expose the log IDevice so the session can reach the device. Measured (pinned, uring, 96 rings, submit-batch 32, fresh 100M keylen16 val96, OFF/ON/OFF bracket): throughput-neutral vs the committed model on both 100% random disk read (t=32 ~6.99M, t=48 ~7.9M; ON bracketed by OFF) and in-memory (t=32 ~52M, t=48 ~85M). 89 DeviceTests pass; GarnetServer builds warning-free. RESP disk serving is managed-CPU-bound, so this is a cleaner, non-poisoning ownership model rather than a throughput lever; kept opt-in pending a broader workload matrix (libaio, mixed, churn soak). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Windows ThreadPoolIoHandler no-ops for new NativeDeviceImpl forwards The branch's native device changes added three methods that NativeDeviceImpl<H> forwards to its handler_ (TryCompleteMine, FlushSubmits, and release_my_ring via ReleaseRing). These were only implemented on the Linux libaio/io_uring handlers. The Windows default (NativeDeviceImpl<ThreadPoolIoHandler>) failed to compile under MSVC once the native-build workflow exercised the Windows RIDs: error C2039: 'release_my_ring' is not a member of 'ThreadPoolIoHandler' Add the three as no-ops on ThreadPoolIoHandler, matching the existing Windows IOCP no-op style: completions fire on threadpool threads (no caller-affine inline drain), submits are immediate (no batch to flush), and io_uring ring ownership is Linux-only (nothing to release). Linux-only file, no effect on the Linux build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 30764320573) from the native sources on 'badrishc/optimize-device-iops'. * Document device completion model + high-latency (cloud) tuning Add a 'Completion model & high-latency (cloud) devices' section to the Device.benchmark README clarifying that the disk-read completion path is block-on-signal (WaitPending parks on the readyResponses SemaphoreSlim; the drainer parks in io_getevents/io_uring_wait_cqe_timeout with io_uring flags=0, i.e. no SQPOLL), not busy-spin. Documents the two poll levers (GARNET_INLINE_DRAIN_AFFINE inline peek; submit-side throttle backpressure) as saturation-only optimizations that fall through to the block path on high-latency devices, plus the cloud tuning knobs (disable inline affine drain; size --throttle-limit to the bandwidth-delay product). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Fix GARNET_SUBMIT_BATCH>=2 read-then-block deadlocks (recover + scan) The opt-in batched-submit device backend (GARNET_SUBMIT_BATCH) defers io_submit until a submitter thread's per-thread read batch reaches its threshold. Any path that issues a sub-threshold burst of device reads and then blocks waiting for their completions without flushing the batch deadlocks: the reads are never submitted, so the completions never fire (drainers idle in io_getevents while the issuer blocks on a never-signaled countdown/semaphore/readyResponses). Two sub-cases, both fixed by flushing the issuing thread's batch: 1. Read-issue-then-block sites (recovery, index, AOF/log scan frames). Flush via IDevice.TryComplete() (flushes before draining on NativeStorageDevice; no-op on immediate-submit devices) right after each read-issue loop, on the issuing thread (the batch is thread-local; async waits may resume on a different thread, so flush at the issue site, not the wait site): - IndexRecovery.BeginMainIndexRecovery (hash-table read; runs first) - MallocFixedPageSize.BeginRecovery (overflow-bucket read) - DeviceLogCommitCheckpointManager.ReadInto (checkpoint metadata read) - AllocatorBase.AsyncReadPagesForRecovery (hybrid-log pages) - AllocatorBase.AsyncReadPageFromDeviceToFrame (scan/iterator frames) - TsavoriteLogAllocatorImpl.AsyncReadPageFromDeviceToFrame (AOF/log scan frames) - ObjectAllocatorImpl object-log read-back sites (truncate, partial-sector flush) The index hash-table read runs first and is a single chunk for a 1g index, so the --recover hang is guaranteed for any batch >= 2. 2. Read re-issued from INSIDE a completion callback (disk hash-chain walk in AllocatorBase.TryVerifyOrReissuePendingRead, reached by the scan-cursor path ScanLookup -> CompletePending(wait:true) that backs RESP SCAN over disk, and by any pending read whose key mismatches on a hash collision). This runs on the completion/drainer thread (or an inline drainer) which returns to a blocking wait without flushing; the re-issued read strands in that thread's sub-threshold batch. CompletePending(wait:true) does flush, but the re-issue happens during its drain (after the flush) and then WaitPending blocks before the next flush -- so the outer flush does not cover it. Fixed with a new flush-ONLY IDevice.FlushSubmits() primitive (submit without draining), called right after the re-issue. Flush-only (not TryComplete) avoids re-entering the completion path from within a completion callback (no recursion on a long chain). Default virtual no-op on StorageDeviceBase; overridden on NativeStorageDevice; no-op on immediate-submit devices. Managed-only; no native change (works with the committed prebuilt .so). Verified: SpanByteIterationPendingCollisionTest (scan-cursor chain-walk) hangs under GARNET_SUBMIT_BATCH>=8 without the fix and passes (~160ms, same as unset) with it across batch {8,32,64}; full SpanByteLogScanTests (10) pass under batch=8 (previously hung the whole process ~75-100s); component recovery tests hang under batch=8 without the recovery flushes and pass with them; full-server --recover reaches Ready across batch {none,8,32,64,1024} and recovers 4.95M keys correctly. Full recovery suite (197) and 89 DeviceTests pass; scan tests (30) pass under batch-unset; GarnetServer Release builds 0-warn; default (non-batching) path is an unchanged no-op poll. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rationalize Native device IO tuning knobs; remove submit batching Distills the device-IOPS optimization study into a small first-class GarnetServer tuning surface and removes the opt-in submit-batching machinery, which was RESP-marginal and the sole source of io_uring ring-ownership ("dead-tid") complexity. Knob surface (all --device-*, wired identically into GarnetServer, KV.benchmark, and Device.benchmark): - Promote io-contexts (ring count) from the GARNET_DEVICE_IO_CONTEXTS env var to a real --device-io-contexts CLI option, plumbed through the LocalStorageNamedDeviceFactory chain to CreateLogDevice. This is the critical io_uring lever (too few rings serialize submitters on a per-ring lock, ~3x slower); libaio is indifferent. - Add --device-queue-depth (per-ring kernel submission depth) and split the old throttle double-duty cleanly into three orthogonal knobs: io-contexts = ring count, queue-depth = per-ring depth, throttle-limit = aggregate in-flight backpressure (<= io-contexts * queue-depth). The hidden min(_,4096) throttle clamp and the throttle/rings ring-depth derivation are removed. Freeze winning behaviors as permanent defaults (delete the env gates): - Affine inline device drain (was GARNET_INLINE_DRAIN_AFFINE) - io_uring batch-reap of completions (was GARNET_URING_BATCH_REAP) - libaio batched io_getevents completion (was GARNET_TRYCOMPLETE_BATCH) Remove entirely: - GARNET_SUBMIT_BATCH + the libaio/io_uring deferred-submit machinery - GARNET_RING_LE_AFFINITY + try_own_ring / release_my_ring / pick_ring_index_le / ring_owner_ ring-ownership state - IDevice.FlushSubmits and the seven batching-driven flush calls injected into the Tsavorite recovery/scan read-then-block paths; those paths revert byte-identical to origin/main, removing the whole latent-deadlock class along with the feature that motivated it. Retains exactly one intentional IDevice addition, TryCompleteMine() (affine inline drain; default = TryComplete(); fixes the context-0 osq_lock storm), and one inert-by-default capacity pair (numIoContexts + queueDepth, default 0 = legacy behavior). Docs: document the 4-knob surface + io-contexts guidance on the config page; canonicalize benchmark option names and refresh the READMEs. 89 DeviceTests pass; GarnetServer Release builds 0-warn; libaio RESP t=48 reqb=1024 reproduces the batch-free floor (7.28M median). Native binaries: linux-x64 rebuilt locally; the remaining RIDs are regenerated by native-build.yml on dispatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 30872979357) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] Right-size NativeStorageDevice NumShards 512 -> 128 Performance-driven right-sizing of the internal per-submitter-thread in-flight sharding constant. A sweep over NumShards {512..16} x threads {32..128} on both Device.benchmark (submitters == threads, the crisp knee) and RESP GET (real GarnetServer, ThreadPool submitters ~35-70) shows throughput is flat down to NumShards ~= the peak concurrent submitter count, with a knee only at NumShards ~= threads/4 (>=4 max-in-flight submitters share one 256-slot free-list => RentSlot spin). 128 gives ~2x headroom over the observed peak submitter count (~70) while trimming the fixed managed slot table from ~4.1 MB to ~1 MB. Every batch-free performance floor is reproduced or exceeded at NumShards=128 (median-of-3, pinned): device 512B libaio 8.82M / uring 8.61M; RESP GET libaio t48 7.41M / t64 7.45M, uring t48 7.41M. 89 DeviceTests pass. Managed-only; no native change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Derive NativeStorageDevice NumShards from ProcessorCount The in-flight shard count must stay >= the peak concurrent submitter count to keep free-list de-contention flat: a knee appears only at NumShards ~ submitters/4, where >=4 max-in-flight submitters share one 256-slot free-list and RentSlot spins. That peak is bounded by the logical processor count, so size NumShards = clamp(2 * ProcessorCount, 128, 1024) instead of a fixed 128 fitted to one machine: - floor 128: the value validated on the sweep hardware (no regression on smaller boxes; ~1 MB fixed managed memory). - 2x ProcessorCount: headroom for transient ThreadPool overshoot under connections >> cores. - cap 1024: bounds the fixed table to ~8 MB on very large machines. Environment.ProcessorCount honors process CPU affinity and cgroup limits, so pinned or containerized servers size to their usable cores. Managed-only; every NumShards use is modulo / loop bound / heap-array size (AssignShard already uses % NumShards), so no compile-time const is required. Builds 0-warn; dotnet format clean; 89 DeviceTests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Harden NativeStorageDevice: fix GPT/Gemini review findings Address cross-environment robustness issues found by the largest GPT (gpt-5.6-sol) and Gemini (gemini-3.1-pro-preview) model reviews of the device-IOPS optimization branch: - Throttle default (highest value): the ctor set ThrottleLimit = 120 (copied from the managed in-box devices), which silently defeated the intended DefaultThrottleLimit (4096). With ~50 active shards that capped out-of-box per-thread in-flight at ~2, throttling the device to ~4-5x below its peak unless the operator passed --device-throttle-limit explicitly. Set the ctor default to DefaultThrottleLimit so the `ThrottleLimit > 0 ? ... : DefaultThrottleLimit` fallback in PerThreadLimit()/init resolves 4096 out of the box. Validated: an out-of-box RESP GET server (no throttle flag) now sustains the ~7.5M libaio peak (t48, reqb1024), matching the explicit-4096 arm. - Dual libaio/liburing repair: extract LoadWithLibaioShim() and route BOTH the primary (Uring) and libaio-only fallback loads through it, so a host that needs the libaio SONAME shim (Ubuntu 24.04+ libaio.so.1t64) AND the liburing2 fallback is repaired regardless of which unresolved SONAME the dynamic loader reports first (previously one ordering dead-ended). - Results-slot clear: ReturnSlot() now clears results[offset] before re-enqueuing the slot, so a completed IO's captured callback delegate and context object are not kept rooted until the slot is next rented (bounded by MaxResults on a mostly-idle device). Cleared before the enqueue so a concurrent RentSlot cannot be clobbered. - ABI probe: the startup native-export probe now also calls NativeDevice_TryCompleteMine so a stale prebuilt .so missing the affine inline-drain export fails fast with the rebuild instruction (it is on the default hot path) instead of an EntryPointNotFoundException mid-run. - AssignShard int-overflow: reduce the round-robin counter modulo NumShards as uint so a long-lived thread-churning server that wraps nextShardSeq past int.MaxValue cannot produce a negative shard index. - Dispose contract doc: document that Dispose() must not be called from inside an IO completion callback (the inline affine-drain path cannot be cheaply detected on the hot completion path), matching the IDevice lifecycle contract. Also clarify the --device-throttle-limit help text and defaults.conf comment (0 => 4096 for the Native device, 120 for the managed in-box devices). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [CI] native-build: verify all load-bearing device exports on every RID The C# NativeStorageDevice loader now hard-probes NativeDevice_NumIoContexts, NativeDevice_QueueRunFor and NativeDevice_TryCompleteMine at device creation (the startup ABI probe; the affine inline-drain path that calls TryCompleteMine is on by default), in addition to NativeDevice_CreateWithBackend bound by the import resolver. A prebuilt binary missing any of these throws at server startup on that RID. The native-build workflow only verified CreateWithBackend, and only on Windows (the Linux job had no export check at all), so a native refactor that dropped one of the now-required exports would pass CI and only fail at runtime on some RID. Add an export-verification step to the Linux job (host nm reads the ELF dynamic symbol table for x64/arm64 and glibc/musl alike) and expand the Windows check to all four symbols. Build-only; does not touch the checked-in binaries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Test] Fix flaky PrimaryUnavailableRecoveryAsync CLUSTERDOWN race The PrimaryUnavailableRecoveryAsync(*, False) cluster tests intermittently failed in CI with "CLUSTERDOWN Hash slot not served" in place of the expected value. During CLUSTER FAILOVER FORCE a replica's role flips to "master" (TryTakeOverForPrimary assigns slots and role) before the failover session clears its recovery flag (EndRecovery). While the promoted primary is still recovering, reads to its own slots are answered with CLUSTERDOWN (ClusterSlotVerify). The test helper UpgradeReplicasAsync only waited for role == master, then immediately issued GETs that raced the still-open recovery window. Wait for LAST_FAILOVER_STATE == "failover-completed" on both promoted replicas after the role flip; that state is set only after the failover session (including EndRecovery) fully returns, so the following reads no longer race recovery. Add a WaitForFailoverCompleted overload keyed by IPEndPoint; the existing int overload delegates to it (behavior-preserving for its callers). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Cap default libaio io_setup reservation to fit stock fs.aio-max-nr libaio's io_setup permanently reserves io-contexts * queue-depth events from the GLOBAL fs.aio-max-nr budget at device creation, whether used or not. The DefaultQueueDepth ceiling (4096, correct for io_uring's per-ring mmap SQ) over-reserves for libaio: a single-ring auxiliary device reserves 4096 events it can never use deeply. In a multi-node cluster process ~15 such auxiliary devices (per-node AOF append, checkpoint bulk IO, replication logs — all default to a single ring) coexist, reserving ~15*4096 = 61440 ≈ a stock 65536 budget, so a transient overlap fails io_setup with errno 11 (EAGAIN). This surfaced as a flaky ClusterFailoverAttachReplicas CI failure ("Native device initialization failed: ... errno 11"). Fix: when --device-queue-depth is left at the default, size the libaio io_setup reservation to the throttle share instead of the io_uring ceiling — NextPow2(2 * ceil(throttle / io-contexts)), floored at 128, capped per-ring at LibaioReservationCap=2048. Deep in-flight should come from MORE rings (higher --device-completion-threads), not one mega-deep ring a lone drainer cannot keep saturated. Multi-ring serving devices (io-contexts >= 4) are unaffected: their 2x throttle share is already <= the cap, so io-contexts * reservation >= throttle and the full aggregate throttle — hence peak IOPS — is preserved. Only low-ring-count auxiliary devices shrink (~15*2048 = 30720, 47% of a stock 65536 budget). An explicit --device-queue-depth is honored verbatim (bypasses this path). Purely managed; no native change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Size default libaio io_setup reservation from fs.aio-max-nr via --device-aio-max-devices Extends the stock-budget reservation cap (30b71bc1) so the default libaio io_setup reservation ceiling is DERIVED from the machine's actual fs.aio-max-nr budget instead of the hardcoded 2048 cap, and exposes the provisioning target as a new knob. libaio io_setup permanently reserves io-contexts * queue-depth events from the machine-global fs.aio-max-nr budget at device creation. To guarantee a known number of Native devices always fit that budget (regardless of how a user sets --device-completion-threads / --device-throttle-limit, and including devices created off the serving factory path such as cluster auxiliary logs and AOF), ResolveLibaioReservationDepth now hard-caps each device's whole reservation (ringCount * depth) at fs.aio-max-nr / AioMaxDevices, halving the per-ring depth (staying pow2) until it fits. AioMaxDevices is a PROCESS-WIDE static (NativeStorageDevice.AioMaxDevices, default 32) because fs.aio-max-nr is a machine-global resource shared by every device in the process, so "how many devices to provision within it" is a process policy, not per-device config. Making it a static also lets devices created via the raw Devices.CreateLogDevice path honor the budget without threading the value through every factory call site. It is applied once from GarnetServerOptions.Initialize() (runs before any device is created) via the new --device-aio-max-devices option (default 32). Behavior is unchanged on both a stock 65536 budget (32 devices -> 2048 events/device, matching the previous cap) and a host that sizes fs.aio-max-nr for its workload (e.g. 4194304 / 32 = 131072/device, which never binds -> serving devices keep full depth, zero IOPS cost). io_uring is unaffected (no global budget; per-ring mmap). Purely managed; no native change. Validation: ClusterFailoverAttachReplicas 8/8 pass at fs.aio-max-nr=65536 (peak aio-nr 32768, 50% margin, no errno-11); device gate libaio 9.12M (>=8.75 floor) / uring control 8.75M (>=8.04 floor) 0 IO errors; RESP serving libaio t48 reqb1024 median 7.46M (>=7.43 floor); 89 DeviceTests pass; new DeviceAioMaxDevicesOption config test; dotnet format clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Harden native device from GPT/Gemini PR review Addresses high-value findings from the GPT-5 and Gemini reviews of this branch. All four are correctness/robustness hardening on the Native device; none change the tuned defaults, and the device + RESP perf gates reproduce the batch-free floors (libaio 8.85M / uring 8.53M device; libaio 7.35M / uring 7.29M RESP t48) with zero IO errors. io_uring submit race (Critical, both reviewers): UringFile::ScheduleOperation treated the io_uring_submit() return count (res >= 1) as proof our tail SQE reached the kernel and released sq_lock around sched_yield during retry. Both are unsafe: io_uring_submit() may partially consume under kernel backpressure (positive count < pending while our SQE is still queued), and dropping the lock lets a peer submitter sharing the ring flush our SQE, after which our own retry misreads the empty SQ as "nothing submitted", rewrites the SQE to a no-op, and frees an io_context whose IO is already in flight -> use-after-free on completion. Fix: hold sq_lock across the whole retry burst and use io_uring_sq_ready(ring) == 0 as the authoritative success signal. Drainers use a separate cq_lock, so holding sq_lock never blocks CQ draining and a transient CQ-full clears as they free space. libaio explicit sub-128 depth: the 3-arg QueueIoHandler ctor floored max_events up to kMaxEvents (128), silently over-reserving from the global fs.aio-max-nr budget whenever the managed layer deliberately passes a shallower depth to fit many coexisting single-ring devices (the --device-aio-max-devices budget math can drive depth below 128). Honor a positive max_events verbatim; only a non-positive value falls back to the default. IDevice.TryCompleteMine default: make it a default interface method delegating to TryComplete() so external IDevice implementations continue to compile and behave correctly without change; sharded devices override it. MaybeReconcileActiveShards: document the known, bounded, self-correcting transient over-subscription (a reused submitter thread seeing a stale-low divisor for at most one reconcile window) and why exact 0<->1 shard-occupancy transition tracking is deliberately avoided (cross-counter race / hot-path lock hazard). Documentation only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Website] Add Device Tuning developer guide Documents the Native storage device tuning surface: every --device-* knob with its default and meaning; the three orthogonal capacity dimensions (ring count N, ring depth D, aggregate in-flight T) and the T <= N*D invariant; the exact derived-parameter formulas (smart io-context default, queue depth, libaio fs.aio-max-nr reservation, effective throttle, per-thread sharding); the internal constants that bound them; precise definitions of headroom vs floor vs cap vs ceiling and why each exists; tuning recipes; and diagnostics. Registered in the Developer Guide sidebar. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Resp.benchmark] Add NVMe RAID-0 sample-results matrix + generator script Adds a "Sample results — 8x NVMe SSD RAID-0" section to the Resp.benchmark README with the full scenario-2 GET throughput matrix ({Libaio,Uring} x {NUMA-pinned,no-pin} x thread-count), measured on out-of-box device defaults (only --storage-tier + --device-io-backend), plus host specs, the fio 8.24M ceiling reference, and repro instructions. Peak ~7.4M ops/sec (uring, pinned, t=48) = ~90% of the fio ceiling driven end-to-end through RESP. Checks in the generator behind the table, benchmark/Resp.benchmark/scripts/nvme-raid0-matrix.sh: sweeps both backends x pin/no-pin x threads, median-of-N per cell, emits the Markdown table. Runs out-of-box defaults by default; set CT/THROTTLE/URING_IOCTX to reproduce the hand-tuned configuration. Measurement + docs only; no product code changed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 30990135225) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] Cap device NumShards at 32 and track activeShards exactly Two device in-flight-accounting refinements in NativeStorageDevice, both managed-only: 1. NumShards = Math.Min(2 * ProcessorCount, 32) (was Clamp(2*cores, 128, 1024)). A fresh device.bench + RESP shard-count ablation (NumShards 448->16 x threads 32->64, both backends) shows throughput is flat from 448 down to ~32 and only dips below ~16 -- the free-list never starves because Throttle() caps each shard's TOTAL in-flight at PerThreadLimit (<= MaxPerThreadInFlight), so a small fixed count neither starves the free-list nor re-introduces counter contention. 32 is ample headroom over the peak concurrent submitter count; the fixed per-shard tables shrink accordingly (MaxResults 8192 max). 2. Maintain activeShards exactly instead of a 200ms background reconcile. Collapse the two monotonic counters (shardSubmitted/shardCompleted) into one signed shardInFlight[] and account activeShards inline: SubmitToShard bumps it on a shard's 0->1 transition, CompleteShard drops it on 1->0, each detected atomically from the interlocked counter's own return value. activeShards is now the exact live occupied-shard count at all times -- it cannot ratchet up under .NET ThreadPool churn, so the per-thread throttle divisor never runs away. Removes MaybeReconcileActiveShards, nextReconcileTicks, ReconcileIntervalMs and the reconcile call on the hot completion path. Validation: core + GarnetServer + Device.benchmark Release build 0-warn; dotnet format clean; 89 DeviceTests pass. Device.bench (512B rand read, /raid, io-ctx32, throttle 4096) matches the ablated ns=32 numbers within noise on both backends (libaio t32 8.63M, uring t32 8.33M), 0 IO errors. RESP GET disk-serving (uring io-contexts 96, t32, successive 15s runs with 25s idle gaps) stays flat at ~6.2M across 7 runs -- the divisor-runaway decline the 200ms reconcile fixed does not return with exact accounting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Resp.benchmark] Make offline MSET serve loop allocation-free and fix flat-buffer NRE The GarnetClientSession and SERedis offline MSET runners allocated a fresh argument array every request: GetRequestArgs() returned the cached payload list, and the runner built `new string[Count + 1]` with "MSET" prepended before each Execute. That is pure hot-loop garbage on the critical serve path. Prepend the command token once, at generation time, instead of per request: - flatRequestBuffer is now List<string[]> and ProcessArgs keeps the parsed command token as element 0, so every cached entry is a complete, ready-to-send argument array ([MSET, k1, v1, ...]). - GetRequestArgs() returns that shared string[] directly. - GCS runner passes it straight to Execute (zero alloc, zero copy, no mutation). InternalExecute serializes synchronously and never retains the array, so sharing it read-only across the run's worker threads is safe. - SERedis runner iterates key/value pairs from index 1 (skips the command token). Also fix a pre-existing NullReferenceException: Run() constructed the ReqGen without flatBufferClient, leaving flatRequestBuffer null and crashing any GCS/SERedis MSET serve phase in GetRequestArgs. Mirror LightOperate and set flatBufferClient for those client types so Generate() populates the cache. Verified: build 0 warn/0 err, dotnet format clean; GCS MSET t=8 runs at ~23M ops/sec with no race or crash; SERedis MSET clean; DBSIZE and GET round-trip confirm correct 16-byte key/value pairs (command token consumed, no off-by-one). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Replace shard-counter stride indexing with a padded ShardCounter struct The per-shard in-flight counters lived in a flat long[] with each shard's counter manually spaced by ShardStride (16 longs = 128 bytes) via `shardInFlight[shard * ShardStride]` at every access site. Replace that with a typed, cache-line-padded struct element, mirroring the existing SpscRingState pattern in the same directory: [StructLayout(LayoutKind.Explicit, Size = 2 * CacheLineBytes)] struct ShardCounter { [FieldOffset(0)] public long InFlight; } shardInFlight becomes ShardCounter[NumShards]; accesses become `shardInFlight[shard].InFlight`. The 128-byte element size preserves the original spacing (each counter owns a cache-line pair, defeating false sharing and the adjacent-line prefetcher), and the generated address math is identical (base + shard*128), so this is a behavior-preserving refactor that removes the error-prone manual stride arithmetic and the ShardStride constant. Validated: - Build 0 warn/0 err; layout check confirms sizeof==128, InFlight 8-byte aligned, elements exactly 128 bytes apart, Interlocked counts exact. - Device.benchmark on 8xNVMe RAID-0 (512B random reads, t=32, throttle 4096): libaio 8.03M vs 8.11M baseline, uring 8.21M vs 8.12M baseline -- both within +/-1.1% (device.bench run-to-run noise), no regression at the ~8M ceiling. - DeviceTests: 89/89 pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Unify TryComplete(mineOnly) + cap BufferPool stripes at 32 Two device/storage changes: 1. Unify IDevice.TryComplete()/TryCompleteMine() into a single TryComplete(bool mineOnly = false) across the managed layer and the native C ABI (NativeDevice_TryComplete(device, int mineOnly); the NativeDevice_TryCompleteMine export is removed). Read paths that await their own ring pass mineOnly:true; the flush-wait keeps walk-all. linux-x64 native binaries (uring + libaio) rebuilt/redeployed; CI symbol checks updated. Other RIDs are refreshed by native-build.yml. 2. SectorAlignedBufferPool stripe count: 128 -> Math.Min(2*ProcessorCount, 32), matching the device NumShards cap. A KV.benchmark scenario-2 sweep (100% random disk reads, 8xNVMe RAID-0) shows a single pool caps at ~45% of peak (ConcurrentQueue cache-line contention) while 32 stripes matches 128 within noise across client-thread counts 32/64/96 on both libaio and io_uring; the knee is throttle-bounded, not thread-bounded. Stripe assignment switched from a pow2 bitmask to overflow-safe uint modulo. Validated: core/KV.benchmark/GarnetServer build 0/0; dotnet format clean; DeviceTests 89/89. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Add opt-in io_uring SQPOLL device knob (--device-uring-sqpoll) Adds io_uring SQPOLL (IORING_SETUP_SQPOLL) as an opt-in device knob so a kernel poll thread drains the submission queue and submissions become syscall-free. All rings share one poll thread (ring 0 spawns it; the rest attach via IORING_SETUP_ATTACH_WQ). Default off; libaio ignores it. Knob chain: native UringIoHandler ctor (sqpoll + sq_thread_idle_ms) -> native_device.h/native_device_wrapper.cc C ABI -> managed NativeStorageDevice P/Invoke + ctor -> Devices.CreateLogDevice + LocalStorageNamedDeviceFactory -> GarnetServer (--device-uring-sqpoll[-idle-ms]) and both benchmarks (KV.benchmark + Device.benchmark). file_windows.h ignores it (Linux-only). Measured DECISIVELY NEGATIVE on this 8xNVMe RAID-0: SQPOLL is ~15-23x SLOWER in every high-IOPS config (Device.bench t=32 8.27M off vs 0.52M on; KV.bench off 7.6M vs on 0.5M) because the single shared kernel poll thread serializes submission (~0.5M/s ceiling) while all submitters spin. Kept as an opt-in knob per request, but the help text, defaults.conf comment and device-tuning doc warn it is not recommended for multi-ring serving and is only useful for low-core / syscall-bound / low-concurrency workloads. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 31075026480) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] SQPOLL: one poll thread per ring + configurable CPU pinning Redesign the opt-in io_uring SQPOLL device knob so each ring gets its OWN kernel submission-poll thread instead of sharing a single one across all rings. The previous shared design (ring 0 spawns the poll thread; rings 1..N-1 attach via IORING_SETUP_ATTACH_WQ) serialized submission through one kernel thread and was a hard throughput ceiling (~15-23x slower on the 8xNVMe RAID-0). Creating every ring with plain IORING_SETUP_SQPOLL restores parallel submission. Add --device-uring-sqpoll-cpus (comma-separated CPU-id list) to optionally pin the poll threads: ring i binds to cpus[i % count] via IORING_SETUP_SQ_AFF; empty (default) leaves them unpinned so the kernel places them freely. Measured (Device.benchmark, 8xNVMe RAID-0, uring, 512B random reads, node0): per-ring SQPOLL now matches or slightly beats the default per-submit path, peaking at 8.39M ops/s (fio parity) at io-contexts=32,threads=32 vs 8.12M without SQPOLL. Leaving the poll threads unpinned (float) is the best default; static pinning is available for isolation but measured slightly worse here. Plumbs --device-uring-sqpoll-cpus through the full chain (native ctors -> C ABI -> managed P/Invoke -> factory -> Device/KV benchmarks -> host config) and updates the device-tuning doc + config test. Rebuilt the linux-x64 native .so (both variants); other RIDs regenerated by native-build.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 31082559176) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] SQPOLL: remove static CPU-pin knob (float is strictly better) The --device-uring-sqpoll-cpus knob (IORING_SETUP_SQ_AFF + sq_thread_cpu per ring) was measured strictly inferior to leaving the poll threads unpinned across every configuration on the 8xNVMe RAID-0 target: at the 32/32 peak, float held 8.39M ops/s while pinning dropped to 8.01M (and to 4.69M at 16/32). With node 0's cores mostly idle the kernel spreads the per-ring poll threads better than any static map, and pinning them onto the submitter/RESP cores costs throughput. Since there is no configuration where pinning wins here, drop the knob entirely rather than ship a foot-gun. Keeps the per-ring design (each ring created with plain IORING_SETUP_SQPOLL, no IORING_SETUP_ATTACH_WQ) which is the actual win. Removes the sqpoll_cpus parameter across the whole chain: native ParseCpuList / sqpoll_cpus_ field / SQ_AFF block (file_linux.h), the ctors (6-arg -> 5-arg in file_linux.h / file_windows.h / native_device.h), the C ABI (native_device_wrapper.cc), managed P/Invoke + ctor + field + log (NativeStorageDevice.cs), the Devices.CreateLogDevice / factory signatures, both benchmarks, and the host Options / GarnetServerOptions / defaults.conf. --device-uring-sqpoll and --device-uring-sqpoll-idle-ms are unchanged. Rebuilt both native .so (uring 2218985 -> 2206593; 18 exports each, no instrumentation). dotnet format clean; GarnetServer/benchmarks/test build 0-warn; GarnetServerConfigTests.DeviceUringSqPollOptions updated and passes; 89 DeviceTests pass; SQPOLL smoke run 0 errors. device-tuning.md updated (dropped the pin column/paragraph; float-only results table + tip). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make PR comments precise: drop dev-history and measurement narration Rewrite device/buffer-pool comments added in this branch to state durable, present-tense technical rationale instead of citing one-off profiling/sweep measurements or an older "legacy" mode: - Replace "profiled at ~13%/~26% CPU", "measured +5-7%", "measured neutral vs 65536", "sweep showed", "~45% of peak", "run-to-run noise across 32/64/96" with the underlying design facts they justify. - Reword "legacy" (default/single-ring) references to describe what the path is rather than that it is old. - Genericize the sample tiered-log path in the RESP matrix script. Comment-only changes; native .so binaries are byte-identical and untouched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 31124512411) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] Right-size buffer-pool stripes to 16 via shared sharding formula Factor the two per-thread de-contention counts — the device in-flight shard count (NativeStorageDevice.NumShards) and the sector-aligned buffer-pool free-list stripe count (SectorAlignedBufferPool.stripes) — onto a single shared sizing formula (ConcurrencySharding.Compute) so they cannot diverge, while giving each its own cap because their contention floors differ in kind: - NumShards keeps cap 32: its floor tracks the peak concurrent submitter count, so below ~32 distinct concurrent submitters collide on a shard and the per-shard in-flight counters and slot free-lists re-contend. - stripes drops to cap 16: its free-list traffic is bounded by the device in-flight throttle rather than the submitter count, so it is thread-count-insensitive and holds peak at a smaller count. RESP disk-serving GET (libaio, pinned, 100M x 128B random reads from a RAID-0 span, reqb 1024, median-of-3) is unchanged within noise at the smaller stripe count: t48 7.43M, t64 7.32M — matching the 32-stripe baseline while halving the buffer-pool free-list array. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Separate TryComplete/TryCompleteMine + address PR review comments Split the unified IDevice.TryComplete(bool mineOnly) into two distinct methods: TryComplete() walks all completion contexts/rings (the safe superset used by the allocator flush-wait), and TryCompleteMine() drains only the caller's affine context/ring (the inline submitter-thread path). TryCompleteMine() is a default interface method delegating to TryComplete(), so external IDevice implementations compile unchanged; NativeStorageDevice and the native libaio/uring handlers override both (TryCompleteBatchFor is renamed TryCompleteMineBatch). Also addresses three PR review comments: - ABI arity: split the native creator into an ABI-stable arity-9 NativeDevice_CreateWithBackend forwarder and an extended arity-11 NativeDevice_CreateWithBackendSqPoll body. The managed wrapper binds the extended symbol, so a stale native library fails fast with a clear rebuild message instead of silently reading uninitialised stack for the SQPOLL args. - Throttle docs: document that per-shard admission is approximate (aggregate in-flight can overshoot by up to ~activeShards); the native ring-full retry is the exact kernel-capacity safety backstop, not the precision of the split. - Test coverage: extend CreateNativeForTest with io-contexts/queue-depth/SQPOLL parameters and add round-trip tests for multi-ring, explicit queue depth, and io_uring SQPOLL. Includes the CI export-verify update (probe CreateWithBackendSqPoll and TryCompleteMine on Linux and Windows), knob doc/help polish, and the rebuilt linux-x64 native binaries; the other RIDs are regenerated by native-build.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Collapse native creator to a single lean export Garnet is the only consumer of libnative_device and ships the binary in-tree, rebuilt from the same commit, so the native ABI never has to stay stable across a version skew. Carry the io_uring SQPOLL parameters directly on NativeDevice_CreateWithBackend (one export) and drop the separate NativeDevice_CreateWithBackendSqPoll body plus the arity-frozen forwarder. The managed P/Invoke binds NativeDevice_CreateWithBackend and calls it directly; the create-time EntryPointNotFound guard is removed because the symbol name is unchanged and the startup probe of NumIoContexts / QueueRunFor / TryCompleteMine already rejects a stale library. CI export-verify drops CreateWithBackendSqPoll. Rebuilt the linux-x64 native binaries; other RIDs are regenerated by native-build.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 31220825798) from the native sources on 'badrishc/optimize-device-iops'. * [Docs] Align device-IO docs with final code (smart io_uring ring default + knob ranges) Correct the benchmark READMEs and configuration reference so they reflect the final shape of the Native device IO tuning surface: - Device.benchmark/KV.benchmark READMEs: io_uring no longer needs a manual --device-io-contexts to reach the ceiling. Document the smart ring-count default min(2 x cores, 64) (floored at the drainer count, decoupled from --device-completion-threads); libaio stays at rings = drainers. Update the uring example to drop the explicit --device-io-contexts and record the out-of-box default-rings result (8.45 M) alongside the under-provisioned (~2.9 M) and explicit-32 (8.00 M) rows. - Resp.benchmark README: reduce the stale scenario-2 "quick" GET table (whose uring rows predated the smart default and understated it) to the libaio rows plus a pointer to the authoritative Sample results matrix. - configuration.md: fill the two blank range cells for --device-uring-sqpoll-idle-ms ([0, 600000]) and --device-aio-max-devices ([1, 4096]) to match their [IntRangeValidation] attributes. Docs only; no code or behavior change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] NativeStorageDevice: exception-safe lazy device creation Harden EnsureNativeDeviceCreated against a partially-initialized device. The native handle (newDevice) is created but only published to the nativeDevice field at the very end, after the completion drainer threads are started. If an exception is thrown in between (e.g. Thread construction/Start under resource pressure, or the ABI-probe EntryPointNotFoundException), the old code left two problems: (a) the native handle leaked -- Dispose observes nativeDevice == IntPtr.Zero and skips NativeDevice_Destroy, so the OS file handle / io_uring rings / libaio contexts are never released; and (b) completionThreads could contain null slots for drainers that were never started, which a later Dispose would NullReferenceException on while joining (foreach ... t.Join()). Wrap the QueueRun probe, drainer spin-up, and the publishing Volatile.Write in a try/catch. On any failure, cancel + join whatever drainers were started (they spin-yield on the still-null nativeDevice field, so cancellation is observed promptly), dispose the token, reset the partial fields, NativeDevice_Destroy the handle, and rethrow. The inner EntryPointNotFoundException handler no longer destroys the handle itself (the outer catch now owns that, avoiding a double-destroy). Also make Dispose's drainer join defensive (t?.Join()). Managed-only; no native ABI change. Found independently by two code-review passes. 94 DeviceTests pass; GarnetServer builds 0-warn; dotnet format clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] NativeStorageDevice: remove redundant GARNET_DEVICE_IO_CONTEXTS env override The io_uring ring count is already fully controllable via the documented `--device-io-contexts` server option (Options.DeviceIoContexts -> numIoContexts constructor parameter), and the unset case is handled by the hardware-aware smart default. The GARNET_DEVICE_IO_CONTEXTS environment variable was a leftover tuning backdoor from the optimization phase that duplicated that control, was undocumented, and wrote to stderr on every device creation when set. Remove it; no functionality is lost. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Config] Mark device-uring-sqpoll-idle-ms as Linux-only (io_uring), matching its companion Addresses PR review: the --device-uring-sqpoll-idle-ms help text and the DeviceUringSqPollIdleMs defaults.conf comment lacked the "Linux-only, DeviceType=Native + io_uring" qualifier that its companion --device-uring-sqpoll already carries. The idle window only applies to io_uring SQPOLL, so it is just as Linux/Native/uring-specific; add the qualifier for consistency. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Group device-type-specific tuning into options objects in CreateLogDevice Addresses PR review (param list getting long; break up by device type). The Devices.CreateLogDevice parameter list had grown to 18 params as Native (libaio / io_uring) tuning knobs were added. Introduce two option objects: - NativeDeviceOptions: IoBackend, NumIoContexts, QueueDepth, UringSqPoll, UringSqPollIdleMs (Native-on-Linux backend tuning). - LocalMemoryDeviceOptions: SegmentSize, RingCapacity. CreateLogDevice now takes these instead of the eight loose device-specific params (18 -> 12). numCompletionThreads stays top-level since it is shared (Native completion drainers and LocalMemory parallelism). Behavior is unchanged: the options fields map 1:1 to the previous params with identical defaults. Updated the three call sites that passed device-specific params (LocalStorageNamedDeviceFactory.Get forwarding, and the two benchmarks' LocalMemory calls); the ~140 common call sites are unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Bundle Native tuning into NativeDeviceOptions in device factory + creator Follow-up to the CreateLogDevice refactor: apply the same "break up by device type" grouping to LocalStorageNamedDeviceFactory and LocalStorageNamedDeviceFactoryCreator, which had the same five loose Native (libaio / io_uring) tuning params (ioBackend, numIoContexts, queueDepth, uringSqPoll, uringSqPollIdleMs). Both constructors now take a single NativeDeviceOptions instead; numCompletionThreads stays top-level (shared). Behavior unchanged (1:1 field mapping, same defaults). Updated the two callers that passed the Native params (Options.GetServerOptions and GarnetServerOptions.Initialize); the ~15 other creator callers pass only common params and are unaffected. Factory.Get() now forwards the stored options object directly to CreateLogDevice. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Adopt main's scalable buffer pool; drop striped pool from this PR PR #2063 merged a scalable origin-return SectorAlignedBufferPool (with a --use-legacy-buffer-pool switch to the legacy per-level ConcurrentQueue pool). That supersedes the striped SectorAlignedBufferPool this PR had introduced, so during the rebase onto main the striping was dropped and BufferPool.cs is taken verbatim from main. This follow-up trims the leftovers: ConcurrencySharding no longer sizes a buffer-pool stripe count (StripeCount removed) — it now sizes only the device in-flight shard count (NumShardCount). Updated the NativeStorageDevice sharding doc comment that referenced the buffer pool's stripe count. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] KV.benchmark: refresh NVMe storage-bound results for the scalable buffer pool Re-measured the storage-bound sweep on the origin-return buffer pool using the command documented in the README (100M x 100B, --log-memory 16m, throttle 4096, completion threads 8, --run-threads-sweep 8,32,64, trimmed mean of 3, 8xNVMe RAID-0). Both the magnitude and the shape of the table changed: KV now peaks at ~7.8 M (~95% of the 8.24 M fio ceiling) instead of ~6.7 M, libaio scales through t=64 rather than falling off, uring peaks at t=32, and NUMA pinning lands within run-to-run noise. Updated the headline figure, the table, and the narrative to match, and noted the io-contexts setting used for the uring rows. Device.benchmark results were re-measured on the same pool and reproduce the documented figures within noise, so that README is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Address PR review: drop dead instrumentation, fix inverted docs, harden SQPOLL wakeup Removes the TSAVORITE_DEVICE_INSTRUMENT environment variable together with the counters it gated (submitCount / completeCount / peakNumPending / submitNanos) and the public GetAndResetStats() that exposed them. GetAndResetStats had no callers anywhere in the repo, so the whole block was dead weight on the submit and completion paths and the last remaining tuning backdoor in this PR; the documented control surface is the --device-* options. Corrects two comments that stated the shard-count invariant backwards. Both claimed the count "must stay at or above the peak concurrent submitter count (roughly 2 x ProcessorCount) ... so it is capped at 32", which is self-contradictory: a cap of 32 cannot enforce a floor of 2 x ProcessorCount. The cap in fact bounds MaxResults and the O(shards) TotalInFlight scan, and submitters beyond the shard count share a shard safely because the per-thread throttle already gates each shard's in-flight. Documents the real aggregate in-flight ceiling. PerThreadLimit clamps at MaxPerThreadInFlight, so device-wide in-flight is bounded by NumShards x MaxPerThreadInFlight = 4096 regardless of --device-throttle-limit. The tuning guide advised raising the throttle to 65536 for extra throughput, which the code cannot honor; that recipe is removed and the ceiling is stated in both the derivation and the sharding section. Retries the SQPOLL wakeup on every negative io_uring_submit return rather than only -EAGAIN / -EBUSY. That enter is issued only when the kernel has flagged the poll thread as parked, so any failure (for example -EINTR from a signal) can leave the SQE published with the poller still asleep; with no later submit to redeliver the wakeup the IO never completes and Dispose's drain-wait hangs. Retrying is safe because liburing recomputes the pending count from the ring. Makes Native_ExplicitIoContexts_DefaultDepth_MultiRing actually exercise multi-ring fan-out. Ring assignment is thread-affine and sticky, so issuing all 128 reads from the single NUnit thread drove exactly one of the eight rings and the test's own comment was false; reads are now submitted from eight threads. Aborts nvme-raid0-matrix.sh when the dataset load fails or comes up short. The script runs without set -e, so a failed load previously fell through to the GET sweep, which would serve in-memory misses and publish a high but meaningless NVMe throughput table. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Address PR review: exception-safe init, SQPOLL kernel guard, honour explicit uring depth Review findings from the end-to-end GPT and Gemini passes over this PR. Native: - io_uring SQPOLL now refuses at init on kernels that lack IORING_FEAT_SQPOLL_NONFIXED (pre-5.11). Such kernels only accept ring-registered descriptors under SQPOLL, so every submission of an ordinary fd would complete EBADF; failing init surfaces an actionable error (errno 95) instead of a silently broken data path. native_device.h lists the new cause in the init-failure message. - An explicit per-ring queue depth is honoured verbatim (apart from the power-of-two rounding io_uring_queue_init requires) instead of being floored at kMaxEvents. This matches QueueIoHandler and the documented --device-queue-depth semantics: a caller that asks for a shallow ring to bound per-ring memory now gets one. - -EINTR is treated as transient in both submit loops. The authoritative "consumed" signal is checked first (io_uring_sq_ready == 0 / io_submit returning 1), so the error branch is reached only when nothing was queued and a retry cannot double-submit. Managed: - The exception-safe region in EnsureNativeDeviceCreated starts at handle creation, so a throwing P/Invoke between creation and the first try block can no longer leak the native handle. The two now-redundant destroy calls are removed. - Corrected the PerThreadLimit and CompletionWorker documentation. Tests: the three native multi-ring cases submit from background threads through a shared helper, so reads actually fan out across rings and contexts rather than all landing on the single NUnit thread's affine ring. Docs: Device.benchmark option help and README sections describing ring-full handling, throttle sharding and SQPOLL now match the implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Benchmark READMEs: state the fio ceiling at both measured block sizes The Device/KV/Resp benchmark READMEs compared Garnet throughput measured with 512 B sector reads against a fio ceiling quoted only at 4 K, without saying the two block sizes are comparable. The array is IOPS-bound rather than bandwidth-bound in this range, so the same fio job yields 8.24 M IOPS at 4 K and 8.20 M IOPS at 512 B; the parity percentages are unchanged, but the READMEs now quote both figures so the comparison is explicit. Also drops the inaccurate "4 KB-class random reads" description of the RESP workload, which reads 128 B records over the array's 512 B sectors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tests] nvme-raid0-matrix: read the DBSIZE reply without blocking The load-verification guard read a fixed 32 bytes from the raw RESP socket, but the ":<n>\r\n" integer reply is shorter than that, so on a host without redis-cli the read blocked forever and the matrix never advanced past its first load. Read a single terminated line with a timeout instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tests] nvme-raid0-matrix: verify the load from the client's op count DBSIZE scans the whole hash index, so on the 100 M-key store it does not reply within the probe window and the guard read back an empty count. The loader already reports the number of ops it pushed, which is the same signal without a server round trip. Route both logs through overridable variables. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Scale the buffer pool depot stripes with the machine Disk-serving RESP GET ran ~31% below the same workload on the pre-rebase tree. A cross-commit A/B (two order-flipped rounds, three passes each, fresh 100M-key load per arm) put the regression at 6.72 M ops/s versus 4.62 M with fully disjoint ranges, and a perf call graph placed ~11.6% of server CPU in unresolved libcoreclr frames under NetworkGET_SG that were absent from the faster arm. Instrumenting the pool located it. Buffer gets split 25% owner-local / 75% shared depot, and the depot's stripes are locked stacks, so ~186 threads drove roughly 8 M Monitor.Enter per second across only 8 locks. The stripe count was fixed at 8 regardless of the hardware, so the number of threads that can enter the depot concurrently did not scale with the machine: sized for a small box it serializes a large one. It now derives from ConcurrencySharding, which already sizes the device in-flight shards this way, rounded up to a power of two so the existing mask indexing still applies and floored at 8 so small boxes keep their current width. The cap is 64, which covers the common concurrency range. A stripe sweep at 48 client threads (three order-rotated rounds of three passes, fresh 100M-key load per run) gives 4.51 / 5.15 / 5.73 / 6.85 / 6.82 M ops/s at 8 / 16 / 32 / 64 / 128 stripes, so throughput is flat from 64 onward there. Because a server drives the depot from roughly one thread per connection, the knee does move out at much higher connection counts: at 128 client threads (~266 server threads) 64 stripes gives 5.85 M against 6.70 M at 256. That degradation is accepted in exchange for the smaller stripe array and shorter miss scan; workloads that sustain far more concurrent threads than the cap trade some throughput for those bounds. Widening does not strand buffers, because a depot miss already scans every stripe of the size class rather than only the caller's; the extra stripes cost about 113 KB per pool and nothing per operation. No other pool constant changes, so the byte budget still bounds retained memory exactly as before. Raising LocalCap alongside this was measured and dropped: at the wider stripe count, 128 / 256 / 512 land at 6.86 / 6.96 / 7.02 M ops/s over three order-rotated rounds of three passes, so the larger caps buy 1.4-2.3% while raising the per-thread hoard bound, which nothing steals from until the thread dies. The stripe count alone restores throughput. The full NVMe matrix on device defaults peaks at 7.36 M ops/s (uring, pinned, t=48), reproducing the published table within 4% on every pinned cell, with the pool's fresh-allocation rate falling to zero. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Trim device docs and comments to first-principles statements Remove narration of prior behaviour, rejected alternatives, and measurement history from the device tuning page, the benchmark README, and the device and buffer pool comments, and cut the restatement that followed the floor / cap / ceiling / headroom definitions. Each remaining statement describes what the code does now and the constraint that shapes it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Harden the native device against review-surfaced failure modes End-to-end review of the device changes (agent plus two independent model reviews) surfaced five real defects. Each is fixed here; measurements below confirm no throughput cost. Use-after-free on the submit path. ReadAsync/WriteAsync bump their shard's in-flight count before the P/Invoke, but that bump is dropped by the completion callback running on a drainer thread. A fast completion can therefore drive the count to zero while the submitting thread is still inside native code -- still to run ~EpochGuard, which touches the device's epoch. Dispose can then observe zero in-flight, join the drainers and destroy the device under the returning submitter. Both entry points now take a second, independently balanced lease around the native call, the same guard the non-IO entry points already get from TryLease. io_uring drainer raced SQ submission on kernels before 5.11. Without IORING_FEAT_EXT_ARG, liburing emulates io_uring_wait_cqe_timeout by taking an SQE, writing a timeout request with a reserved user_data and flushing the SQ -- from the completion side, without sq_lock, while submitters mutate the same SQ. The reserved user_data is also non-null, so the batch dispatcher would have treated it as a caller context. The feature bit is now sampled at init and the drainer polls the completion queue instead when it is absent. SQPOLL wakeup could be dropped. Once io_uring_submit publishes an SQE to an SQPOLL ring the kernel owns it, so a failed enter means only that the wakeup was not delivered to a parked poll thread; with no later submit on that ring the IO never completes. The retry now backs off through sched_yield into bounded 1 ms sleeps rather than giving up after a few yields. Thread-start failure leaked the native device. The drainer slot was published before Start(), so a failure under resource pressure left an unstarted thread in the array; the cleanup path's Join() then threw ThreadStateException, escaped, and skipped the destroy. The slot is now published after the thread is running and the destroy runs in an unconditional finally. A transient startup probe error published a device with no drainers. NativeDevice_QueueRun doubles as the capability probe: Windows IOCP returns a permanent negative, but a Linux backend can return a transient negative when the probing thread is interrupted by a signal, which the runtime does routinely. It is now retried before concluding the backend has no drainable queue. Also: clamp RoundUpPow2 at IORING_MAX_ENTRIES so an out-of-range depth through the C ABI cannot overflow; guard the completion callback's own logger call so a throwing host logger cannot defeat the drainer firewall; and warn when a libaio reservation cannot be brought within its per-device share of fs.aio-max-nr, since depth cannot fall below one event per ring. The AioMaxDevices help text and docs no longer claim an unconditional guarantee. Device.benchmark, libaio, 512 B random reads on 8x NVMe RAID-0, 32 threads, 6 interleaved samples per arm: 8.469 M ops/s with the fixes against 8.527 M before them, with fully overlapping ranges. The two extra interlocked operations land on a shard line the submitter already owns. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Align the libaio budget cap description with its actual guarantee The `--device-aio-max-devices` section claimed the per-device reservation cap guarantees at least that many devices can be created "regardless of the other knobs". The cap cannot go below one event per ring, so a device configured with more rings than its per-device share still exceeds it, and the budget is the machine total rather than what remains after other processes. The derivation section and the option help text already state both limits; this makes the knob section agree and links to the derivation. Also record the second reservation warning in the many-devices recipe: one fires when `N x D` exceeds `fs.aio-max-nr`, the other when a device cannot be brought within its per-device share. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Correct the libaio reservation floor description The floor glossary entry claimed the reservation depth is never sized below 128, but the per-device AIO-budget clamp runs after the floor and halves past it when the whole-device reservation does not fit. On a stock 65536 budget, 32 io-contexts resolve to a depth of 64. State the precedence in both the doc and the constant's XML doc: the budget ceiling overrides the floor, because exceeding the budget fails device creation while a shallow ring only costs throughput. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Record that the AIO-budget ceiling can reduce the effective throttle The libaio reservation notes claimed multi-ring serving devices keep the full aggregate throttle at no IOPS cost. That holds for the throttle-share math, but the per-device fs.aio-max-nr ceiling runs last and caps effectiveThrottleLimit at ringCount * depth. On a stock 65536 budget that bound is 2048, halving the default 4096 throttle at every ring count. Qualify the claim in the doc and in the three matching code comments, and give the operator the sizing rule: fs.aio-max-nr / --device-aio-max-devices >= throttle. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] State the slot free-list headroom in terms of the shard Throttle() gates a shard's whole in-flight against a limit clamped to MaxPerThreadInFlight, so a shard holds at most that many slots however many submitter threads share it. The SlotsPerShard summary attributed the bound to a single submitter instead, which reads as if the free-list could be drained by several threads sharing one shard. Match the framing already used by Throttle() and RentSlot(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Scope the reservation share-math claim to the share clamps The consequences list stated the no-IOPS-cost property unconditionally and retracted it three bullets later, so a reader on a stock budget -- where the ceiling always binds -- takes away the wrong default. Attach the property to the share clamps that provide it and point forward to the ceiling that runs after them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 31969160992) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] Drop two unreachable retry paths from the native device An audit of the review-driven hardening found two of its retry budgets guard failure modes that cannot occur. The startup QueueRun probe retried on the premise that a Linux backend can answer with a transient negative when a signal interrupts the probing thread. A zero timeout never blocks, so neither backend can: libaio passes a zero io_getevents timeout and io_uring reads the completion queue in user space without a syscall. A 3,000,000-iteration harness that hammered io_getevents with a zero timeout while another thread signalled the caller through a handler installed without SA_RESTART observed zero negatives and zero EINTR. The probe is back to a single call. The io_uring SQPOLL wakeup path gained a second backoff stage of 1000 one millisecond sleeps. An enter carrying only IORING_ENTER_SQ_WAKEUP never waits, so it cannot return EINTR, which io_uring_enter(2) documents only for IORING_ENTER_GETEVENTS; every other error on that path is permanent. Sleeping cannot turn such a failure into a success, and sq_lock and the epoch are held throughout, so the stage only held the ring for a second before reaching the same outcome. The bounded yield budget is restored. Both comments now state the mechanism that actually applies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 31975517483) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] Bound NativeStorageDevice.Dispose's in-flight drain Dispose waited for in-flight IOs with an unbounded `while (TotalInFlight() != 0) Thread.Yield();`. In-flight only returns to zero if the kernel completes every accepted IO, so a completion that is never delivered spins there forever: an unkillable teardown that pins a core and reports nothing. Lost completions are reachable from several directions — a stalled device or driver, a dropped CQE, or an io_uring ring whose SQPOLL thread has died, after which the ring accepts submissions whose completions never arrive. The drain now runs against a deadline. It sits orders of magnitude above any legitimate drain: outstanding IOs are already queued in the kernel, and every native call a lease is held across is individually bounded (the submit paths unwind after a fixed yield budget, QueueRunFor takes a timeout), so reaching it means completions are lost rather than slow. On expiry the count is logged and teardown proceeds, which is safe because the drainers are cancelled and joined before the handle is freed — no user callback can run during teardown — and NativeDevice_Destroy cancels or waits for whatever the kernel still owns. SpinWait replaces the bare Thread.Yield() so the normal microsecond drain stays spin-fast while a drain that runs to the deadline does not pin a core. Verified by injecting a phantom in-flight count with no matching completion: before, Dispose did not return within 150s; after, it returns at the deadline. file_linux.cc: comment only. The SQPOLL submit path noted that a later submit redelivers the wakeup, which holds only while the poll thread is parked. Once it is gone nothing redelivers. Failing those IOs individually would not restore correctness — everything already in flight on that ring is lost with it, and the kernel holds the only reference to their contexts (their user_data), so there is nothing to enumerate. The comment now states that and points at the drain bound. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Resp.benchmark: republish the NVMe matrix from verified-load runs The published RESP matrix was measured by a version of the generator that ran the GET sweep unconditionally after the MSET load, without checking that the load had written the dataset. A key that was never written is answered from memory with no device IO, and a miss is roughly 30x cheaper than a disk read, so a partial load inflates the reported figure: on this array an unloaded store reports over 150 M ops/sec against a true storage-bound 6.3 M, and a ~35% shortfall alone accounts for the previously published 7.36 M. Re-measured every cell with the load verification in place (the loader reports 99,876,864 of 100 M ops). The pinned peak is 6.96 M (uring, t=48) rather than 7.41 M, both backends now peak at t=48 instead of libaio climbing through t=64, and the no-pin rows drop further, which widens the NUMA-pinning gap. The new figures are also physically coherent with the neighbouring layers, which the old ones were not: raw device ~8.4 M > KV ~7.8 M > RESP ~7.0 M. Also tighten the generator's own short-load guard from 90% to 99% of DBSIZE. The 90% floor still admitted an ~11% overstatement; at 99% the reported figure is within ~1% of the fully-loaded value. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] KV.benchmark: correct the NVMe storage-bound results The published storage-bound table was measured against a build whose buffer pool differs from the one this branch ships, so it does not reproduce. Every row is replaced with a fresh trimmed-mean-of-3 measurement on the current tree, and the peak claim drops from ~7.8 M (95% of fio) to ~6.3 M (77%). The uring rows now use the smart ring default instead of an explicit --device-io-contexts 32, which under-provisions at t=64; the default is faster in every cell and makes the table match the documented command. Also corrects two claims the new data contradicts: NUMA pinning is within noise for the disk-bound scenario (it gates the RAM-served scenarios), and the three benchmarks use different datasets, so their numbers do not form an ordering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Size the buffer pool's owner-local chain to a thread's IO pipeline SectorAlignedBufferPool keeps a per-(thread, class) chain of returned buffers and spills to a lock-guarded depot once the chain reaches LocalCap. A caller that rents many buffers before returning any therefore hits the depot for everything past the first LocalCap rents, however small its actual working set. KV.benchmark issues --batch-size (default 1024) reads per iteration before draining, so its per-thread burst is 1024 buffers against a LocalCap of 128: 7 of every 8 rents and returns went through the depot. Instrumenting the pool during a disk-bound run measured 147.4M depot pops against 49.3M local hits (74.8% of gets) and 147.6M depot spills against 196.9M local pushes, with zero cross-thread and zero large-class traffic - about 1.09B Monitor operations, 6.6 per KV operation. User CPU per operation was 2.92x the raw device path (3.011us vs 1.033us) while kernel CPU per operation was lower, so the cost was managed-side, not IO. Raising LocalCap to 1024 admits the whole burst. Both consumers improve: KV.benchmark, 100M x 100B on 8xNVMe RAID-0, 100% random reads from disk, trimmed means of 3 (ops/sec): backend pin t=8 t=32 t=64 libaio node-0 2,366,537 6,861,507 7,707,696 libaio none 2,391,726 6,903,713 7,650,788 uring node-0 2,296,988 7,621,662 7,226,697 uring none 2,266,429 7,722,915 7,368,255 Peak 7.72M against 6.34M before, +21.8%, and 94% of the array's 8.24M fio ceiling. Sweeping LocalCap alone at t=32 traces the burst exactly: 128 -> 5.738M, 512 -> 6.786M, 1024 -> 6.942M, 2048 -> 6.948M, 4096 -> 6.910M, i.e. throughput follows min(LocalCap / 1024, 1) and flattens once the cap covers the batch. RESP GET, 3 rounds with the arms rotated each round, medians: LocalCap t=48 t=64 128 6.725M 6.423M 1024 7.398M 7.216M 2048 7.229M 7.182M The 128 and 1024 ranges are disjoint at both thread counts. 2048 is below 1024, so 1024 is the value both consumers want. Peak RSS falls: 9,135,776 kB at 128 against 8,930,012 kB at 1024. At the lower cap the depot overflows and the pool drops and re-allocates buffers continuously (dropped-because-full tracked fresh allocations one for one); covering the burst collapses the fresh-allocation rate to near zero. The chain is intrusive - it links through the buffer's own next pointer - so a larger cap costs no structural memory, and LocalByteCap (32 MB per thread and class) remains the bound that stops one thread parking the budget. This reverses the 2026-08-15 measurement that kept main's 128. That study swept 128/256/512 and read +2.32% at 512, inside the "under 5%, keep main's setting" band. It was under-ranged: the RESP scatter-gather path rents about 2000 buffers per network batch, so 512 covers only a quarter of the burst. The arms that cross the burst threshold are worth 10-12%, well outside the band, so the same rule now selects the change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Bound the buffer pool's owner-local retention in bytes The pool budgets bytes, but the owner-local chain was bounded by a count. A count cannot bound a byte budget: one size class's buffer is up to 512x another's, so the same count means 512x the bytes depending on which class a thread happens to use. The count a thread actually needs is its IO pipeline depth, which is a property of the caller, not of the pool. The mismatch had a reachable consequence. `permitBytes` is reserved for the life of a parked buffer, so per-(thread, class) retention was min(LocalCap x classBytes, LocalByteCap). Summed over the 16 small classes at LocalCap 1024 that is 302 MB against a 256 MB small sub-budget, so a single thread could exhaust it. Once `TryReserve` fails, every thread runs non-cacheable - every Get allocates and every Return frees - and nothing trims the thread holding the bytes. Replace both caps with a per-thread byte ceiling: the small sub-budget divided by `ConcurrencySharding.ExpectedConcurrentThreads` (min(2 x cores, 64)), floored at 1 MB. A thread's classes share that ceiling work-conservingly, so a single-class thread gets all of it. At the ceiling, `TryMakeRoom` reclaims from the class furthest above its equal share (max-min fair) and refuses the request only if the requester is already at or above its own share, which also makes self-eviction impossible. Victims are spilled to the depot, not dropped, so they stay allocated, budgeted and reusable. `ThreadShard.activeClasses` is maintained incrementally so the common single-class case tests the ceiling in O(1) - this matters because a thread at steady state sits at the ceiling and enters that path on every return. Worst-case per-thread retention drops from 302 MB to 4 MB here (75x). The smaller ceiling does not increase churn, because the real bound is the caller's IO pipeline depth and both workloads sit under the slice (KV ~1.15 MB, RESP ~2.3 MB derived from their burst sizes and class mix). Measured on 8xNVMe RAID-0, pinned, median-of-3, against the count-cap arm: RESP libaio t=48/64 7.094/7.095 (was 6.923/7.167), uring 7.389/7.030 (was 7.291/7.013); KV libaio t=64 7.76 M; peak RSS 8,927,844 kB (was 8,930,012 kB). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Resp.benchmark: refresh the NVMe storage-bound matrix The published figures were measured before the buffer pool's owner-local retention was sized to a thread's IO pipeline, so they understated libaio by up to 7% (pinned t=48: 6.51 -> 6.97 M) and reported the wrong peak thread count for libaio. Re-measured the full 16-cell out-of-box matrix with the generator script (median of 3, load verified at 99,876,864 of 100 M keys) and republished every cell: backend NUMA t=8 t=32 t=48 t=64 Libaio srv-0/cli-1 1.82 5.70 6.97 7.06 Libaio no pin 1.38 5.09 5.72 5.57 Uring srv-0/cli-1 1.82 6.13 7.32 6.89 Uring no pin 1.48 4.60 5.55 6.29 Derived claims corrected along with the numbers: - Peak is 7.32 M (uring, pinned, t=48), ~89% of the array's fio ceiling, not 7.0 M / ~84%. - The backends no longer peak at the same thread count: uring peaks at t=48 and eases off at t=64, while libaio is still climbing at t=64. The guidance is now to sweep the t=48-64 band. - The backend-parity bullet quantifies the spread (uring leads 5-7% at t=32-48, libaio by 2% at t=64) instead of claiming "within a few percent", which the t=32 cell no longer supports. - The tuned-vs-default claim drops its numeric bound, which was not re-measured on this binary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Resp.benchmark: correct the backend-gap range The published matrix shows uring ahead of libaio by 7.5% at t=32 and 5.0% at t=48, and libaio ahead by 2.5% at t=64. State 5-8% and ~2% so the bullet matches the table above it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Buffer pool: describe the per-thread byte cap The website design doc still described LocalCap (128 buffers) and LocalByteCap (32 MB) bounding a single (thread, size-class) local stack. Neither constant exists: local retention is now bounded in bytes per thread, shared across that thread's size classes, at smallBudget / ExpectedConcurrentThreads floored at MinThreadLocalBytes (4 MB at the 1 GiB default), with max-min fair admission across classes via TryMakeRoom. Also correct the summary's '8-way' depot striping, which contradicted section 6 (8-64 stripes, sized from the processor count), and note that a spill relocates a buffer to the depot with its permit intact rather than making it uncacheable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Docs] Buffer pool: state the design rather than defend it The depot section justified its lock against ConcurrentStack point by point, and the large-class section closed with before/after reuse, allocation and RSS figures from an earlier iteration. Both read as defenses of past decisions rather than a description of the design. Keep every substantive fact - atomic close, exact capacity bound, allocation-free push, and why large classes have no per-thread locality to exploit - and state them as properties of the final design. Align the equivalent source comment in ReturnOriginReturn. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Report a permanent io_uring SQPOLL wakeup failure On the SQPOLL submit path the retry loop re-enters io_uring_enter until the wakeup is delivered or the yield budget is exhausted. On exhaustion the entry is already published to the kernel, so the operation is reported submitted and the loop's last errno was discarded, leaving a ring that can accept IOs whose completions never arrive with no signal to the operator. Emit the errno and its consequence once per device, claimed through an atomic flag so a ring that fails for every submission reports a single line rather than one per IO. The report does not change the submitted outcome: the entry is kernel-owned and the kernel holds the only reference to the affected contexts via their user_data, so there is nothing to enumerate and rewriting the SQE would race the poll thread. NativeStorageDevice.Dispose already bounds its drain, so a ring in this state cannot hang teardown. Rebuilt the linux-x64 uring binary. The libaio variant is unchanged: this code is inside the FASTER_URING guard, which that build does not define. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 32199655122) from the native sources on 'badrishc/optimize-device-iops'. * [Tsavorite] Gate native handle destruction on native-call leases; fix two benchmark option bugs Addresses the fresh Copilot review on #2018. NativeStorageDevice.Dispose(): the shard in-flight counter was overloaded for two things with different termination properties — IOs awaiting a completion (which can be lost forever, hence the bounded drain) and leases held by threads executing inside native code. TryComplete/TryCompleteMine hold a lease across a native call that dispatches user callbacks inline, so a lease is bounded only by user code, not by the "every native call is bounded" claim the drain comment made. A lost completion could therefore trip the deadline while a native frame was still running, and the subsequent NativeDevice_Destroy would free its rings and locks underneath it. Leases are now counted separately (a second field on the existing padded shard counter, so no extra cache miss) and handle destruction waits on that counter after the in-flight drain. Leases are bumped after in-flight and dropped before it, so leases <= in-flight always holds and the normal path costs one extra read. No new lease can be acquired once disposedFlag is published, so the wait only covers calls already in native code; if it does expire the handle is leaked rather than freed, which is bounded and diagnosable where a use-after-free is not. Device.benchmark: --device-throttle-limit had no effect on LocalMemory. LocalMemoryDevice does not override StorageDeviceBase.Throttle() (which returns false), so its in-flight bound is the per-submitter SPSC ring, and RingCapacity was left at 0 => 1024. Map the throttle onto the ring capacity as KV.benchmark already does, and print the resolved value. Resp.benchmark: --load-threads 0 reached DbSize % loadDbThreads and crashed with DivideByZeroException; reject values below 1 up front. file_linux.cc is comment-only: state that io_uring_sq_ready() is sqe_tail-khead, measured against the kernel head rather than ktail, so submit's flush cannot drive it to zero and it remains an exact "kernel consumed our SQE" test under short submits and failed enters. No behavior change, so the prebuilt binaries are unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Tsavorite] Document the t_shards/slotIndex/registry relation in BufferPool Add a section comment explaining how the two indexes over the (pool x thread) shard matrix relate: t_shards is the thread-side index (ThreadStatic, strong refs, hot path), the registry is the pool-side index (weak refs, cold path), and slotIndex is the recyclable key joining them -- which is what makes the identity check on every read necessary. Rename registry -> poolShardRegistry (and its lock, threshold, and compaction helper) so the pool-side index is distinguishable from the thread-side one at each use site. * [Tsavorite] Fix wording in slotIndex comment for consistency --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> | 18 天前 | |
Helm chart improvements (#1802) * fix(chart): remove token and add automountServiceAccountToken Signed-off-by: babykart <babykart@gmail.com> * feat(chart): add keys to parametrize IPv4/IPv6 behavior Signed-off-by: babykart <babykart@gmail.com> * style(chart): move livenessProbe & readinessProbe keys Signed-off-by: babykart <babykart@gmail.com> * feat(chart): add values.schema.json Signed-off-by: babykart <babykart@gmail.com> * fix(chart): add missing ipFamily keys to service-headless Signed-off-by: babykart <babykart@gmail.com> * fix(chart): fix volumeMounts[].mountPath for config-volume Signed-off-by: babykart <babykart@gmail.com> --------- Signed-off-by: babykart <babykart@gmail.com> Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> | 2 个月前 | |
Remove sync-over-async where possible, consolidate blocking into helpers, add analyzers (#1714) * knock out some of the 'easy' .Result uses * convert some easy GetResult() calls to async * proper conversion of a lot of migration code to async * convert cluster epoch polls to async, and everything that's downstream of that * async some of checkpointing and replication * remove some more .Result, mostly by shifting to await helper methods * all .Results that can be removed (or turned into .GetAwaiter().GetResult()) have been removed * another audit of .GetResult(); converting more to tasks where appropriate * formatting * standardize on GetAwaiter().GetResult(); propogates exceptions correctly, but also more unique for searching * move all .GetResult()'s to a helper for easier auditing; cleanup more 'could be async' code * remove explicit .Wait() calls where possible, switch to helper where not * adopt Microsoft.VisualStudio.Threading analyzers; fix or suppress all findings * address feedback * fix nit * change NetworkHandler.Start() so auth proceeds asynchronously - introduce IsAuthenticated(...) to allow polling for auth completion, which many callers assume | 4 个月前 | |
Fixes around CU copies and delete deadlocks (#2082) * with recovery cleanup, blocking during delete request is unnecessary - stressed tests to confirm * update vector-sets.md | 13 天前 | |
Fix Allure wiring: remove duplicate attributes and fix CI check for transitive dependencies (#1784) * Split cluster tests into 5 parallel CI projects Split Garnet.test.cluster into separate projects to enable parallel CI: - Garnet.test.cluster: shared infra + basic cluster tests (144 tests) - Garnet.test.cluster.migrate: migrate + slot verification (67 tests) - Garnet.test.cluster.replication: all replication tests (372 tests) - Garnet.test.cluster.vectorsets: vector set cluster tests (30 tests) - Garnet.test.cluster.multilog: sharded log replication tests (163 tests) Child projects reference base via ProjectReference + InternalsVisibleTo. Updated CI and nightly workflow matrices to run all 5 in parallel. Fixed Allure wiring check to find AllureTestBase in referenced assemblies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Split replication tests into TLS, AsyncReplay, and DisklessSync projects Separate Garnet.test.cluster.replication into 3 additional projects for parallel CI execution: - Garnet.test.cluster.replication.tls: TLS replication tests (inherits ClusterReplicationBaseTests with useTLS=true) - Garnet.test.cluster.replication.asyncreplay: Async replay tests (inherits ClusterReplicationBaseTests with asyncReplay=true) - Garnet.test.cluster.replication.disklesssync: Diskless sync tests (standalone ClusterReplicationDisklessSyncTests) Base replication project retains ClusterReplicationBaseTests and ClusterResetDuringReplicationTests. Updated InternalsVisibleTo, multilog project references, CI and nightly workflow matrices, and solution file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Split Garnet.test into 8 projects for parallel CI execution Split the monolithic Garnet.test project into 8 focused test projects: - Garnet.test (base): RESP core, config, admin, infra (~774 tests) - Garnet.test.collections: Hash, List, Set, SortedSet, Geo (~746 tests) - Garnet.test.acl: ACL and auth tests (~426 tests) - Garnet.test.scripting: Lua, custom commands, transactions, AOF, modules (~585 tests) - Garnet.test.complexstring: Bitmap, HyperLogLog (~386 tests) - Garnet.test.vectorset: VectorSet tests (~36 tests) - Garnet.test.rangeindex: RangeIndex tests (~58 tests) - Garnet.test.extensions: JSON, DiskANN, revivification, storage internals (~527 tests) Each child project references Garnet.test for shared infrastructure (TestUtils, AllureTestBase, extensions). InternalsVisibleTo entries added to Garnet.server, Garnet.host, GarnetServer, GarnetJSON, and Tsavorite.core for child projects. Updated CI and nightly workflows with expanded test matrices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add missing [AllureNUnit] attributes to cluster sub-project test fixtures Added [AllureNUnit] and [TestFixture] to ClusterReplicationTLS, ClusterReplicationAsyncReplay, ClusterReplicationShardedLog, and ClusterReplicationDisklessSyncShardedLog to pass CI Allure wiring check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Allure wiring: remove duplicate [AllureNUnit] and fix CI check for transitive dependencies - Remove duplicate [AllureNUnit] from derived test classes (TLS, AsyncReplay, MultiLog) that inherit it from base classes, fixing runtime error 'Unable to change the container context because the test context is active' - Fix CI Allure wiring check to search AppDomain.GetAssemblies() instead of using Assembly.Load() on direct references, which failed for AllureTestBase in transitive dependencies (e.g. Garnet.test.cluster via Garnet.test.cluster.replication) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: add NuGet package cache to all CI jobs Add actions/cache@v4 for ~/.nuget/packages to format-garnet, format-tsavorite, build-test-garnet, and build-test-tsavorite jobs. Cache key is based on runner.os and hash of *.csproj and Directory.Packages.props files, with a fallback restore-key for partial matches. This reduces dotnet restore time, especially on Windows runners where install dependencies averaged 1.1 min (up to 4.1 min) per job. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: split Garnet build and test into separate jobs Split build-test-garnet into two jobs: - build-garnet: 8 jobs (os × framework × config) that restore, build, and upload bin/obj artifacts with 1-day retention - test-garnet: 128 jobs that download build artifacts and run dotnet test --no-build --no-restore This eliminates 120 redundant builds (~316 CPU-minutes per run). Test jobs no longer need Rust toolchain, NuGet cache, or restore steps. The pipeline-success job now depends on test-garnet instead of build-test-garnet. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Split Tsavorite tests into 5 subprojects for parallel CI execution Split the monolithic Tsavorite.test project into 5 test subprojects to enable parallel CI execution and reduce the critical-path bottleneck: - Tsavorite.test (core tests, shared infrastructure) - Tsavorite.test.recordops (revivification, delete/dispose, record lifecycle) - Tsavorite.test.session (session, unsafe context, read cache chain tests) - Tsavorite.test.hlog (log, scan, device, spanbyte, compaction tests) - Tsavorite.test.recovery (recovery, checkpoint, object recovery tests) CI workflow changes: - ci.yml: Split build-test-tsavorite into separate build-tsavorite and test-tsavorite jobs with 5x4 matrix (os x framework x config x subproject) - nightly.yml: Added new subprojects to test matrix with proper allure-results staging and cleanup paths - Scoped NuGet restore to Tsavorite.slnx only in build job Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Cache Azurite npm install and skip for subprojects that don't need it - Add npm cache (actions/cache) for Azurite in both ci.yml and nightly.yml - In ci.yml, conditionally skip Node.js setup, Azurite install, and RunAzureTests env var for test.recordops, test.session, and test.recovery (only test and test.hlog use Azure storage device tests) - Saves ~200s on Windows / ~20s on Linux per skipped subproject job Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Allure wiring: restore path prefix on Tsavorite artifact download upload-artifact strips the common ancestor (libs/storage/Tsavorite/cs/) from the glob paths. Specify path: on download-artifact to restore the prefix so the Allure check and test run steps find DLLs at the expected location. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add -graph flag to dotnet build for faster CI builds MSBuild graph build constructs the project dependency graph upfront and schedules builds optimally, avoiding redundant project evaluations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Decouple format checks from build dependency chain Run format-garnet and format-tsavorite in parallel with builds instead of blocking them. Format failures still gate pipeline-success, but builds and tests can start immediately, saving ~1-2 min off the critical path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix test execution: run dotnet test against DLLs directly dotnet test with --no-build against project directories silently exits with 0 tests when MSBuild evaluation fails to locate the pre-built assemblies. Running against the DLL directly bypasses MSBuild entirely and reliably discovers and executes tests from downloaded artifacts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore execute permission on GarnetServer after artifact download upload-artifact strips the Unix execute bit. Add chmod +x for the GarnetServer binary on Linux after downloading build artifacts, so tests that launch it as a subprocess (e.g., GarnetBitmapTests) work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Split test.session into test.session + test.session.context Split the Tsavorite.test.session subproject (23.1 min worst case) into two for better CI parallelism: - test.session: ReadCacheChainTests, ReadAddressTests, NativeReadCacheTests, RandomReadCacheTests (read-cache focused) - test.session.context: TransactionalUnsafeContextTests, UnsafeContextTests, SessionTests, FunctionPerSessionTests (session/context focused) Extracted shared helpers into test/SessionContextTestUtils.cs to resolve cross-project dependencies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Tsavorite formatting in SessionContextTestUtils.cs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove lowMemory param from ClusterSRPrimaryCheckpointRetrieve Make manySegments imply lowMemory instead of requiring both parameters. This reduces test combinations from 16 to 8 (removes redundant cases where lowMemory=false,manySegments=true which was already a no-op due to the existing manySegments = lowMemory && manySegments guard). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reorganize test projects into standalone/ and cluster/ subdirectories Move Garnet.test* projects under test/standalone/ and Garnet.test.cluster* projects under test/cluster/ for better organization. - Move 8 standalone test projects to test/standalone/ - Move 8 cluster test projects to test/cluster/ - Update all .csproj relative paths (ProjectReference, Compile Include, EmbeddedResource, AssemblyOriginatorKeyFile, testcerts) - Update Garnet.slnx with nested folder structure - Update TstRunner.csproj project references - Update CI workflows (ci.yml, nightly.yml) with subdir resolution - Update runGarnetTests.cmd path Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Consolidate Tsavorite test projects under single test/ directory Move satellite test projects (test.hlog, test.recordops, test.recovery, test.session, test.session.context) from cs/ into cs/test/ as subdirectories. - Move 5 satellite test project directories under cs/test/ - Update satellite csproj relative paths (src/, Garnet.snk, Tsavorite.test ref) - Add Compile Remove glob to Tsavorite.test.csproj to exclude subdirectories - Update Tsavorite.slnx project paths - Fix Tsavorite.test.csproj AllureTestBase.cs path for prior Garnet test move - Update CI workflows (ci.yml, nightly.yml) Tsavorite directory maps Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove Rust toolchain setup from CI build-garnet job Pre-built native BfTree binaries exist for all platforms (linux-x64, win-x64, osx-x64, osx-arm64). The BfTreeInterop.csproj already gracefully skips cargo when it's not installed, falling back to pre-built binaries. Rust is only needed to rebuild from source for bftree/range-index development. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove all Allure test reporting infrastructure - Remove AllureTestBase class, rename to TestBase (keep RunningTests tracking) - Remove [AllureNUnit] attribute from all 158 test fixtures - Remove Allure.Net.Commons and Allure.NUnit package references from all csproj files - Remove Allure packages from Directory.Packages.props - Remove Allure wiring verification steps from ci.yml (Garnet + Tsavorite) - Remove Allure CLI install, results staging, report generation from nightly.yml - Remove Allure artifact/history/report steps from deploy-website.yml - Delete test/Allure/ directory (GenerateAllureReport.ps1, categories.json) - Update docs: README badge, copilot-instructions, skills, onboarding Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CI: remove -graph flag and artifact pipeline for test jobs The -graph flag on dotnet build silently skips projects inside nested <Folder> elements in .slnx files. After restructuring test directories into test/standalone/ and test/cluster/ with nested solution folders, only 4 of 16 test projects were built, causing 'DLL not found' errors in 33 of 36 test jobs. Changes: - Remove -graph flag from build-garnet and build-tsavorite jobs - Remove artifact upload/download pipeline between build and test jobs - Test jobs now build inline via dotnet test <project-dir> (matching the pattern used on main branch) - Add NuGet cache and restore to test jobs - Remove redundant framework dimension from build-garnet matrix - Keep build jobs as fast-fail compilation gates Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix RootTestsProjectPath for restructured test directories After moving test projects into test/standalone/ and test/cluster/, Split("Garnet.test")[0] returns test/standalone/ instead of test/. Navigate up one level to reach the correct test/ root directory. Fixes DocsTests, RespCommandTests, RespCustomCommandTests, and RespModuleTests failures caused by incorrect relative path resolution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix refs to test for vs * fix config test * adjust max parallel jobs in CI * revert parallel max cap * split workflow graph --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> | 3 个月前 | |
[dev] Remove sync-over-async where possible, consolidate blocking into helpers, add analyzers (#1730) * knock out some of the 'easy' .Result uses * convert some easy GetResult() calls to async * proper conversion of a lot of migration code to async * convert cluster epoch polls to async, and everything that's downstream of that * async some of checkpointing and replication * remove some more .Result, mostly by shifting to await helper methods * all .Results that can be removed (or turned into .GetAwaiter().GetResult()) have been removed * another audit of .GetResult(); converting more to tasks where appropriate * formatting * standardize on GetAwaiter().GetResult(); propogates exceptions correctly, but also more unique for searching * move all .GetResult()'s to a helper for easier auditing; cleanup more 'could be async' code * remove explicit .Wait() calls where possible, switch to helper where not * adopt Microsoft.VisualStudio.Threading analyzers; fix or suppress all findings * fixup a few blocking calls that slipped through; fix formatting * address feedback * fix nit * change NetworkHandler.Start() so auth proceeds asynchronously - introduce IsAuthenticated(...) to allow polling for auth completion, which many callers assume * formatting | 4 个月前 | |
Fix CI BDN allocation flakiness (#1989) * [Test] Fix BDN allocation-gate flakiness for Set/JSON benchmarks Two independent causes of intermittent BDN allocation-gate failures: 1. Server GC with concurrent (background) collections corrupts BenchmarkDotNet's process-wide MemoryDiagnoser: a Gen2 GC in the measurement window makes the per-op allocation swing wildly (0 to 8x). Use blocking (non-concurrent) GC so the measurement is deterministic. Fixes the JSON ModuleJsonGetCommand spikes. 2. SAddPopSingle emptied its set every iteration (SPOP removes the last member), so the key was deleted and recreated, churning the object-store log and allocating 16MB pages that intermittently landed in the measurement window. Seed key1 with a keeper and use deterministic add/remove so the set is never emptied and the ops stay in-place (no log growth). Renamed to SAddRemSingle since it now exercises SADD/SREM, and tightened the SAddRem/SAddRemSingle thresholds to reflect the ~10x lower, stable allocation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [RESP] Reduce JSON.GET allocation, near-zero for the root path JSON.GET allocated ~536 B/op, dominated by: JSONPath parse of "$" (160), LINQ Sum + boxed WriteBulkString (176), a fresh SerializeToUtf8Bytes byte[] (144), a new List<byte[]> (32), and a path string (24). The high gen0 churn also fed the allocation-measurement variance. - Add a root ("$") fast path (GarnetJsonObject.TryGetRoot) that serializes the whole document into thread-static reusable ArrayBufferWriter/Utf8JsonWriter and writes straight to the RESP output, skipping the path string, JSONPath evaluation, the per-item byte[], and the List. - Add a non-boxing WriteBulkString(ReadOnlySpan<byte[]>) overload in RespWriteUtils/RespMemoryWriter and use it from the JSON GET reader, removing the LINQ Sum and boxed IEnumerable enumeration (helps all JSON.GET paths). ModuleJsonGetCommand: 53600 -> 2400 B (-95.5%, ~24 B/op); latency 143 -> 70 us. Non-root paths (deep/array/filter) drop 5-21% from the WriteBulkString change; their remaining cost is in the custom JSONPath SelectNodes. All 44 JSON command tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Test] Mark Set AOF allocation gates warn-only The AOF variants of SAddRem/SAddRemSingle allocate identically to the None and ACL variants (net10 3200 B, net8 6400 B); the AOF enqueue path adds no per-op heap allocation. Under Server GC the MemoryDiagnoser can occasionally over-count one GC allocation-context quantum (~18 KB) when a background collection lands in the AOF measurement window, producing rare spikes (e.g. 21699 B) unrelated to the workload. Keep None/ACL hard-gated at 6400 and make the AOF variants warn-only, matching the existing treatment of the larger noisy AOF set operations in this config. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Test] Stabilize SortedSet pop/remove BDN benchmarks ZPopMax, ZPopMin, ZMPop and ZRemRangeByScore ran against a single-member sorted set, so each iteration emptied and recreated the key, churning the object-store log (~48-113 KB per op, with rare 16 MB log-page spikes up to ~520 KB). Give each its own key seeded with a keeper member "k" scored so the benchmarked command removes the re-added "d" but never "k", keeping the set non-empty and the operation in-place. Local net10 None: ZPopMax/ZPopMin 12000 B, ZMPop 20000 B, ZRemRangeByScore 48000 B. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Test] Right-size over-provisioned object-type BDN warn thresholds Lower the warn-only allocation thresholds for Hash, ZAddRem and the newly stabilized SortedSet pop/remove benchmarks to reflect measured allocations (~2x the stable value) instead of the previous 10-20x over-provisioning (e.g. HSetDel_None 120000->19200, HMSet_None 114000->12800, ZAddRem_None 149000->32000, ZPopMax_None 48001->24000). The _AOF thresholds for ZPopMax/ ZPopMin keep headroom (74000) for the net10 AOF MemoryDiagnoser over-count; denied-path ACL keys stay at the 6400 baseline. Validated against CI run 30191658574 results: no hard fails, no new warnings on the adjusted benchmarks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Test] Make Set add/remove allocation gates warn-only on all params SAddRem/SAddRemSingle allocate ~3200-6400 B but, being object-store RMW benchmarks, occasionally over-count by ~1 GC allocation-context quantum (~17-18 KB) under Server GC when a background collection lands in the measurement window (observed net10 None 20755 B, net10 AOF 21699 B). The over-count can hit any param, so a tight hard gate flakes. Make the None and ACL variants warn-only to match the already-warn-only AOF variant and the object-store benchmark convention (Hash/List/SortedSet are all warn-only). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Test] Set object-type BDN warn thresholds to expected measured values Replace the ~2x/headroom placeholders with the actual measured allocations after stabilization (the gate adds its own +10% tolerance): SortedSet ZAddRem 15200, ZMPop 20000, ZPopMax/ZPopMin 12000, ZRemRangeByScore 48000; Hash HSetDel 9600, HMSet/HMGet 6400, HSetNx/HIncrby/HStrLen 3200, HScan 776. Denied-path ACL keys stay at the 6400 baseline. These are warn-only gates, so the intermittent net10 Server-GC MemoryDiagnoser over-count now surfaces as a warning rather than being masked by an inflated threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Test] Restore SAddPopSingle as a stable SADD+SPOP BDN benchmark Revert the earlier SAddPopSingle->SAddRemSingle rename, which had turned a distinct SPOP benchmark into a near-duplicate of SAddRem (both SADD+SREM) and dropped SPOP coverage. Instead keep SADD+SPOP but make it stable: each SADD uses a distinct member (a pool of batchSize members, far larger than the set) so a re-added member has always been popped already and the set never shrinks, and key1 is seeded with several keepers so it never empties. With the key never deleted/recreated there is no object-store log churn, giving a deterministic 7200 B/op across all params and both frameworks (measured; verified stable over repeated runs). Add +spop to the benchmark ACL grant so the ACL variant runs the pop in-place rather than leaving SADD to grow the set unbounded. This preserves the SAddPopSingle chart series (no rename) and keeps SAddRem (SREM) and SAddPopSingle (SPOP) as distinct operations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [RESP] Fix JSON.GET length overflow and unbounded root-buffer retention Two issues in the JSON.GET fast path, found in review: - The response length was summed in an unchecked int, so a >2GB result wrapped negative and bypassed the RESP writer bounds check (the prior LINQ Sum was overflow-checked). Sum in long and reject results above int.MaxValue. - The thread-static root-GET ArrayBufferWriter only grows; ResetWrittenCount keeps the backing array, so one large JSON.GET permanently inflated a long-lived session thread's buffer. Release the buffers once they exceed a 64 KB retention cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Test] Make SAddPopSingle deterministic; tighten BDN gate coverage - SAddPopSingle: replace the 100-distinct-member SPOP batch with `SADD key a b; SPOP key`, which adds two members and pops one so the set deterministically oscillates between 2 and 1 members and never empties -- no reliance on pool-size-vs-set-size probability. Stable 10400 B (net8) / 7200 B (net10), and faster than the distinct-member version. - Config: SAddPopSingle -> 10400 (covers net8); fix two dead gate keys whose `WARN-ON-FAIL_expected_` prefix never matched the parser (LPushPop_ACL, SScan_AOF); ModuleJsonGetCommand -> warn-only 2400 (was hard 360000, which no longer detects regressions after the near-zero root-path optimization). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Test] Correct BDN GC-job comment and note Workstation-GC follow-up The prior comment claimed Server GC keeps the allocation measurement deterministic; review showed the Server-GC allocation quantum still intermittently inflates MemoryDiagnoser (notably on .NET 10), which is why the object-store allocation gates are warn-only. Note the Workstation-GC job as the follow-up that would allow deterministic hard gates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Test] Make ModuleJsonGetRecursive allocation gate warn-only ModuleJsonGetRecursive is a recursive-descent JSONPath query that genuinely allocates ~30-47 MB with large run-to-run variance (it passed earlier runs below the 33 MB hard-gate ceiling and exceeded it at 47 MB in another). A tight hard gate on a benchmark that variable produces false-positive CI failures, so make it warn-only. The large baseline itself comes from the per-node allocations in the custom JSONPath engine (non-root paths) -- reducing that is a separate follow-up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Objects] JSON.GET: write single-path results directly, cutting per-node allocations The single-path JSON.GET path built a List<byte[]> with one byte[] per matched node, then concatenated them into the RESP bulk string. For a recursive wildcard ($..*) matching ~181 nodes this dominated allocation (BDN ModuleJsonGetRecursive = 2.42 MB/op locally, and up to 47 MB under Server-GC over-count on CI, causing allocation-gate flakes). Add GarnetJsonObject.TryGetToWriter: evaluate the JSONPath and serialize the matched nodes as a single JSON array straight into the thread-static buffer/writer already used by the root ("$") fast path, then write once as a RESP bulk string. Eliminates the List and the per-node byte[] arrays. Output is byte-identical (verified: 44 JsonCommandsTest cases incl. $..author multi-node, wildcard, filter, and empty-match paths). Result (net10 None, local): ModuleJsonGetRecursive 2.42 MB -> 0.52 MB (-78%, deterministic across runs) and 6.20 ms -> 4.59 ms (-26%). The residual 0.52 MB is the lazy-iterator JSONPath engine (SelectNodes/SelectMany) -- a separate follow-up. Also apply the reviewer nit: the residual formatted/multi-path length guard now checks Array.MaxLength (the real byte[] cap) instead of int.MaxValue. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Test] Right-size JSON.GET allocation gates to post-fix expected values After the JSON.GET direct-writer change, re-baseline the JsonOperations gates to the measured post-fix allocations (max of net8/net10; the gate applies +10%): - ModuleJsonGetRecursive 50000000 -> 521600 (real 521600, was 2421600 pre-fix) - ModuleJsonGetDeepPath 600000 -> 44000 (HARD -> warn; was 13x over-provisioned) - ModuleJsonGetArrayPath 370000 -> 53600 (HARD -> warn; was 7x over) - ModuleJsonGetFilterPath 770000 -> 64800 (HARD -> warn; was 12x over) - ModuleJsonGetArrayElementsPath 800 -> 800 (HARD -> warn; unchanged value) All JSON.GET path gates are warn-only, matching the object-store policy: the allocation MemoryDiagnoser over-counts under Server GC when a background GC lands in the measurement window. Data (CI runs 30221983810 clean, 30219601330 spike) shows this is proportional to per-op allocation volume: only the heavy Recursive ($..* = ~181 nodes) spikes (net10: 15.4M/47.1M vs 2.42M base); the light ops stayed dead-stable even in the spike run. Cutting Recursive's real allocation 4.6x shrinks its spike ceiling proportionally; the residual net10 spikes now warn (non-blocking) instead of failing. A deterministic hard gate needs the deferred Workstation-GC job (noted in Program.cs). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Test] Use Workstation GC for BDN so allocation is measured accurately Root cause of the roving BDN allocation-gate flakes (FilterPath 61k->407k, Recursive->47M, and the object-store spikes): under Server GC, BDN's MemoryDiagnoser reads GC.GetTotalAllocatedBytes(precise:false), whose counter includes the unused allocation-context budgets across Server GC's per-core heaps. When a gen1/gen2 GC resets those contexts between BDN's before/after readings, the counter jumps by N-heaps x budget -- a pure measurement artifact that roves across whichever benchmark catches a context reset mid-measurement. Evidence: spiked "GC: 25 1 1 1667621296 4096" (407134 B/op) vs clean "GC: 32 0 0 250675200 4096" (61200 B/op) for the same benchmark in different runs. Fix: run the benchmarks under Workstation GC (single heap, small budget), which makes GetTotalAllocatedBytes accurate/deterministic. Verified locally: allocation is byte-identical to the clean Server-GC value (FilterPath 61200, Recursive 521600, ZAddRem 15200, SAddRem 3200, HSetDel 6400) and the spikes cannot occur. These are single-client benchmarks, so Server GC provided no latency-fidelity benefit; measured latency is equal-or-slightly-better under Workstation GC (charts are alert-only and this is an improvement, so no alert). This enables restoring tight hard allocation gates (follow-up: re-baseline gates from a Workstation-GC CI run). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Test] Restore hard allocation gates for benchmarks stabilized by Workstation GC With Workstation GC, MemoryDiagnoser now measures allocation deterministically (CI run 30233233102: each of these benchmarks reported an identical value across all 16 rows -- 2 OS x 2 runtimes x 4 params -- with zero spikes). Convert the verified-stable, substantial benchmarks from warn-only back to hard `expected_` gates so real allocation regressions block CI again: - JSON GET: Command 2400, DeepPath 44000, ArrayPath 53600, ArrayElements 800, FilterPath 64800, Recursive 521600 - Set: SAddRem 6400, SAddPopSingle 10400 - SortedSet: ZAddRem 15200, ZMPop 20000 (None/AOF), ZPopMax/ZPopMin 12000 (None/AOF), ZRemRangeByScore 48000 (None/AOF) - Hash: HSetDel 9600; List: LPushPop 11200 (was 14400) Thresholds are the max of the net8/net10 measured values; the gate adds +10%, so the deterministic values leave headroom and cannot false-fail. Left warn-only: the ACL-denied pop paths (NOPERM), the churn benchmarks that empty+recreate a key (ZRemRangeByLex ~112k, ZRemRangeByRank ~86k -- genuine allocation variance, not a measurement artifact), and the near-zero read ops (many now correctly report "-" under Workstation GC, which the gate treats as 0). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Test] Address PR review: tighten BDN comments - Program.cs: shorten the GC comment to a present-tense, factual statement of why Workstation GC is used (drop transient spike figures and narrative). - SetOperations.cs: SAddPopSingle comment now states the invariant it relies on (the set is never emptied and key1 is never deleted/recreated) instead of the inaccurate "oscillates between 2 and 1 members" claim (key1 is also seeded with a keeper and shared with SAddRem). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a * [Test] Convert deterministic object-store gates from warn-only to hard Now that Workstation GC makes MemoryDiagnoser deterministic, warn-only is no longer needed for benchmarks with stable allocation. Evaluated every warn-only Set/SortedSet/Hash benchmark against TWO independent Workstation-GC CI runs (30233233102, 30236831459): 62 are deterministic (within-runtime spread <=3% in both runs) and are converted to hard `expected_` gates at the measured max (the gate's +10% leaves headroom). Many old warn thresholds were 5-13x over-provisioned because they absorbed the Server-GC over-count (e.g. ZCount 150000->16800, ZRange 320000->24800) -- the new hard gates are both tight and accurate. Kept warn-only: - Genuine run-to-run variance (churn that empties+recreates a key): ZRemRangeByLex, ZRemRangeByRank, ZRangeStore, and the near-zero oscillator HIncrbyFloat. - Pure-read ops that allocate 0 (show "-"): a hard gate at 0 has no absolute headroom, so these stay warn-only to avoid false-fails on any measurement blip. RawString(LTM)/Lua/Script warn gates are left as-is (their results were not part of this evaluation). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e66a8a41-4525-473a-9095-a5e614d95e1a | 1 个月前 | |
Harden LightEpoch: make the epoch announce part of the slot-claim CAS (#2015) * Fix unfenced epoch announce in LightEpoch (x86-64) A thread entering a protected region announced its epoch with a plain store, which is not ordered against the reclaimer's later load of the same slot. A reclaimer could scan a live reader's slot, see it as free, raise SafeToReclaimEpoch past the reader's epoch, and free a page the reader was about to dereference. The claim CAS now writes localCurrentEpoch directly, so claiming the slot and announcing the epoch are one locked RMW and the announce is globally visible before any load in the protected region can issue. No barrier and no atomic is added; the lock cmpxchg was already there. localCurrentEpoch doubles as the ownership word, sound because a protected thread never announces epoch 0. Release() correspondingly clears threadId before freeing the slot. LightEpoch moves to its own Garnet.LightEpoch project so it can be tested and disassembled in isolation, with unit tests and a quarantine litmus harness under playground/LightEpochLitmus. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 38cd2f3a-d460-407c-8a96-a7330974ce99 * Build LightEpochLitmus for net8.0 as well CodeQL builds the whole solution with 'dotnet build -f net8.0', which failed because the litmus project only targeted net10.0. Inherit the repo default net8.0;net10.0 and pin the Dockerfile/README commands to net10.0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b5f062b9-2d72-4cf0-b6f3-4c9beb98d068 * Rename LightEpoch projects to Tsavorite.epoch / Tsavorite.test.epoch Drop the Garnet prefix from the epoch library and its unit test project so they match the Tsavorite.core / Tsavorite.test.* naming of the rest of the storage engine. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d306b783-4a33-4673-9e29-790995df8179 * Move LightEpoch back into Tsavorite.core Undo the split of LightEpoch into a standalone project: the sources return to src/core/Epochs/ and the duplicated Murmur3 helper is dropped in favor of the existing Utility.Murmur3. LightEpochLitmus now references Tsavorite.core. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d306b783-4a33-4673-9e29-790995df8179 * Drop redundant IDisposable declaration from LightEpoch LightEpoch already exposed a public Dispose(); declaring the interface adds nothing and was not part of the fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d306b783-4a33-4673-9e29-790995df8179 * Restore .github/copilot-instructions.md to match main Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d306b783-4a33-4673-9e29-790995df8179 * Document ProtectAndDrain refresh semantics and address review feedback Rewrite the ProtectAndDrain docs to state that it refreshes an already-held slot rather than entering the protected region, that SafeToReclaimEpoch is gated on the minimum announced epoch so a holder that never refreshes stalls reclamation process-wide, and that a refresh relinquishes protection for the previously announced epoch. Read CurrentEpoch volatile when announcing and inline ReserveEntry into ReserveEntryForThread. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b6f65ef0-7c7f-40d9-a95f-3ecb62de9d5a * Trim ProtectAndDrain doc comment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b6f65ef0-7c7f-40d9-a95f-3ecb62de9d5a * Simplify ProtectAndDrain doc wording Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b6f65ef0-7c7f-40d9-a95f-3ecb62de9d5a * Inline the announced epoch read in ProtectAndDrain Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b6f65ef0-7c7f-40d9-a95f-3ecb62de9d5a * Restore the original Drain argument expression in ProtectAndDrain Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b6f65ef0-7c7f-40d9-a95f-3ecb62de9d5a * Use volatile accesses for the announced-epoch word Make the ProtectAndDrain announce a release store and the reclaimer's ComputeNewSafeToReclaimEpoch scan an acquire load, so a slot's announced epoch cannot be observed out of order with the work it guards. Hoist the announced epoch into a local so Drain reuses it instead of re-reading the slot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8b507896-826f-444c-b958-51c19863f429 --------- Co-authored-by: Tiago Napoli <tiagonapoli@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 38cd2f3a-d460-407c-8a96-a7330974ce99 Copilot-Session: b5f062b9-2d72-4cf0-b6f3-4c9beb98d068 Copilot-Session: d306b783-4a33-4673-9e29-790995df8179 Copilot-Session: b6f65ef0-7c7f-40d9-a95f-3ecb62de9d5a Copilot-Session: 8b507896-826f-444c-b958-51c19863f429 | 1 个月前 | |
Simplify etags [dev] (#1739) * Add SETWITHETAG command (Phase 1 of ETag refactoring) Add new SETWITHETAG command as a dedicated top-level command for setting key-value pairs with ETags. This replaces the SET ... WITHETAG option pattern and is the first step in corralling ETag semantics to dedicated commands only. Changes: - Add RespCommand.SETWITHETAG enum value - Add SlowParseCommand parsing and CmdStrings constant - Add NetworkSETWITHETAG handler in BasicEtagCommands.cs - Add dispatch in RespServerSession.ProcessArrayCommands - Add RMW callbacks (NeedInitialUpdate, InitialUpdater, InPlaceUpdater, NeedCopyUpdate, CopyUpdater, PostInitialUpdater) - Add VarLenInputMethods cases (GetRMWInitialFieldInfo, GetRMWModifiedFieldInfo) with HasETag=true - Add SupportedCommand.cs entry - Add GarnetCommandsInfo.json and GarnetCommandsDocs.json entries - Add generated resource JSON entries - Add SetWithEtagACLsAsync ACL test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make all non-ETag commands ETag-blind (Phases 2-6) Remove WITHETAG option from SET and RENAME/RENAMENX commands. Make all non-ETag commands completely ETag-blind — they do not read, check, update, or remove ETags. Phase 2: Remove WITHETAG from SET - Remove WITHETAG parsing from NetworkSETEXNX - Remove EtagOption enum entirely - SET paths always pass withEtag: false Phase 3: Remove WITHETAG from RENAME/RENAMENX - Remove withEtag parameter from API signatures - Simplify RENAME to strictly 2-arg commands Phase 4: Make RMW methods ETag-blind - Remove hadETagPreMutation/shouldUpdateEtag for non-ETag commands - ETag state only initialized for SETIFMATCH/SETIFGREATER/SETWITHETAG/DELIFGREATER - Remove ETag logic from SET/SETKEEPTTL/SETEXNX InitialUpdater/InPlaceUpdater/CopyUpdater - Remove custom command ETag rejection - Reader only sets ETag state for GETWITHETAG/GETIFNOTMATCH Phase 5: Make UpsertMethods ETag-blind - MainStore/ObjectStore/UnifiedStore UpsertMethods pass false for inputHasETag Phase 6: Make UnifiedStore ETag-blind - Remove shouldUpdateEtag pattern from UnifiedStore RMW/InPlaceUpdater - Clean up VarLenInputMethods to not use CheckWithETagFlag Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clean up remaining ETag artifacts (Phase 8) Remove dead ETag constants and error strings: - Remove WITHETAG CmdString (no longer parsed by any command) - Remove RESP_ERR_ETAG_ON_CUSTOM_PROC error string - Remove RESP_ERR_WITHETAG_AND_GETVALUE error string Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update tests and samples for ETag refactoring (Phase 9) - Replace all SET...WITHETAG with SETWITHETAG in tests and samples - Remove tests for removed features: SET WITHETAG NX/XX/KEEPTTL, RENAME WITHETAG, custom command ETag rejection - Remove tests verifying ETag auto-increment on non-ETag commands (APPEND, INCR, DECR, SETRANGE, SETBIT, BITFIELD) - Remove tests verifying SET strips ETags (undefined behavior now) - Fix ETag state initialization for ETag commands on keys without ETags - Update samples/ETag Caching.cs and OccSimulation.cs All 83 ETag tests pass. All 345 RespTests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update documentation for ETag refactoring (Phase 10) - Replace SET (WITHETAG) with SETWITHETAG in garnet-specific.md - Rewrite 'Compatibility with Non-ETag Commands' section with prominent key partitioning warning and usage examples - Remove WITHETAG from SET syntax in raw-string.md - Remove WITHETAG from RENAME/RENAMENX in generic-commands.md - Update blog post intro to reference dedicated ETag command set Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove ETag from Tsavorite hot paths & extract ETag RMW to NoInline helpers (Phases 6b, 7) Phase 6b: Remove dead ETag weight from Tsavorite - Remove ETag field from RecordMetadata struct (never consumed) - Remove eTag field from PendingContext - Remove ~20 pendingContext.eTag assignments from InternalRead/RMW/Upsert/Delete - Simplify RecordMetadata construction in AllocatorScan, TsavoriteIterator, TsavoriteThread, CompletedOutput, Tsavorite.cs Phase 7: Extract ETag RMW logic to NoInline helpers - Create RMWMethods.Etags.cs with 9 [NoInlining] helper methods - Each ETag case in InPlaceUpdaterWorker/InitialUpdater/CopyUpdater/NeedCopyUpdate becomes a single-line call, keeping hot-path methods compact for JIT - Helpers: HandleDelIfGreater*, HandleSetIfMatch*, HandleSetWithEtag* for InPlaceUpdate, InitialUpdate, CopyUpdate, and NeedCopyUpdate Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Extract ETag Reader logic to ReadMethods.Etags.cs with NoInlining Move GETWITHETAG and GETIFNOTMATCH handling from Reader into a [NoInlining] HandleEtagReader helper in ReadMethods.Etags.cs. ETag commands are now delegated at the beginning of Reader before any other processing, keeping the hot-path Reader method compact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove dead inputHasETag parameter from InPlaceWriter helpers All callers pass false — remove the parameter entirely from InPlaceWriterForSpanValue, InPlaceWriterForHeapObjectValue, InPlaceWriterForLogRecordValue, and rename UpdateExpirationAndETag to UpdateExpiration (ETag logic removed). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make ETag RMW handling fully stateless; remove ETagState Replace functionsState.etagState with local long existingEtag passed to each handler. Consolidate ETag command dispatch into single HandleEtagInPlaceUpdateWorker / HandleEtagCopyUpdateWorker / HandleEtagNeedCopyUpdate dispatchers that read the ETag once and delegate to individual NoInline helpers. - Remove ETagState struct and EtagState.cs entirely - Remove etagState field from FunctionsState - Remove PostInitialUpdater ETag reset - CopyRespWithEtagData now takes long etag directly - Remove isEtagCommand/shouldUpdateEtag variables from hot paths - All ETag logic is now stateless — no shared mutable state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move ETag dispatch into switch jump tables (Read, RMW) Replace separate if-checks before switches with cases inside the switch for ETag commands. This lets the JIT generate a single jump table covering both ETag and non-ETag commands, eliminating an extra branch on the hot path. - InPlaceUpdaterWorker: ETag cases in the switch - CopyUpdater: ETag cases in the switch - Reader: restructured to use switch with ETag cases Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove withEtag parameter from NetworkSET_Conditional The withEtag parameter is dead — the flag it sets (SetWithETagFlag) is never read by any RMW callback. RMW methods dispatch on the RespCommand enum value instead. Replace the parameter with isEtagCommand derived from the cmd inside the method. Also remove the dead SetWithETagFlag() call from DELIFGREATER handler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Separate ETag API from non-ETag SET_Conditional - Add SET_ETagConditional and DEL_ETagConditional to IGarnetApi/GarnetApi as dedicated ETag API methods, separate from SET_Conditional - Split NetworkSET_Conditional (non-ETag only) from ExecuteETagSetCommand (shared by SETWITHETAG/SETIFMATCH/SETIFGREATER) - Remove SetWithETagFlag() and CheckWithETagFlag() from RespInputHeader - Remove RespInputFlags.WithEtag enum value - Remove stale ETag comments from ObjectStore VarLenInputMethods - Custom procedure test uses SET_ETagConditional with SETWITHETAG Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix trailing newline in RMWMethods.Etags.cs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix SETWITHETAG TTL semantics and format SETWITHETAG without EX/PX now clears any existing expiration, matching SET semantics. Previously it would preserve the old TTL. - InPlaceUpdate: remove expiration when input.arg1 == 0 - CopyUpdate: don't carry forward srcLogRecord.Expiration - VarLenInputMethods: SETWITHETAG doesn't preserve source expiration (split from SETIFMATCH/SETIFGREATER which do preserve) - Fix trailing newline in RMWMethods.Etags.cs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Optimize VarLenInputMethods: don't reserve ETag space for non-ETag commands Change GetRMWModifiedFieldInfo default from HasETag=srcLogRecord.Info.HasETag to HasETag=false. Non-ETag commands (INCR, APPEND, SETRANGE, etc.) no longer allocate ETag space in copy-updated records. ETag commands explicitly set HasETag=true in their own switch cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments - Fix SETWITHETAG CopyUpdate to clear TTL when no EX/PX (match SET semantics) - Fix stale comment in PrivateMethods.cs about ETag value layout - Update doc wording: 'do not preserve' instead of 'do not remove' - Add regression test: SetWithEtagClearsTTLWhenNoExpiryProvided - Keep using Tsavorite.core (needed for PinnedSpanByte in ExecuteETagSetCommand) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> | 4 个月前 | |
Fixes for Vector Set `RENAME`s in AOF replay (#2080) * Failing tests for Vector Set renames followed by an AOF restore * fix set flags replay from AOF * stopgap commit; bigger problem with transactions uncovered * restore tests * AOF recovery requires VectorManager be quiescent at various points, so add those blocks * ensure VADD setflags and VADD createindex are sequenced relative to each other; special case renames when copying a Vector Set as we need to ensure context and index pointer consistency there * formatting * address feedback * use matching contexts for rename special casing | 17 天前 | |
Fixes around CU copies and delete deadlocks (#2082) * with recovery cleanup, blocking during delete request is unnecessary - stressed tests to confirm * update vector-sets.md | 13 天前 | |
Improve dockerfiles (#224) * Improve dockerfiles * Set docker image distro as ubuntu * Add Alpine based Dockfile * Remove line breaks * Add .dockerignore * Simply .dockerignore * Fix arch issue for nano server * Fix nanoserver docker build issue * Revert nanoserver dockerfile --------- Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> Co-authored-by: Irina Spiridonova <irinasp@microsoft.com> | 2 年前 | |
Remove sync-over-async where possible, consolidate blocking into helpers, add analyzers (#1714) * knock out some of the 'easy' .Result uses * convert some easy GetResult() calls to async * proper conversion of a lot of migration code to async * convert cluster epoch polls to async, and everything that's downstream of that * async some of checkpointing and replication * remove some more .Result, mostly by shifting to await helper methods * all .Results that can be removed (or turned into .GetAwaiter().GetResult()) have been removed * another audit of .GetResult(); converting more to tasks where appropriate * formatting * standardize on GetAwaiter().GetResult(); propogates exceptions correctly, but also more unique for searching * move all .GetResult()'s to a helper for easier auditing; cleanup more 'could be async' code * remove explicit .Wait() calls where possible, switch to helper where not * adopt Microsoft.VisualStudio.Threading analyzers; fix or suppress all findings * address feedback * fix nit * change NetworkHandler.Start() so auth proceeds asynchronously - introduce IsAuthenticated(...) to allow polling for auth completion, which many callers assume | 4 个月前 | |
[Tsavorite] Default to NativeStorageDevice on x64 Linux (+ segment-boundary flush and write-error propagation fixes) (#1991) * Switch default Linux device to NativeStorageDevice Make DeviceType.Native the default on Linux (previously RandomAccess); Windows already defaulted to Native. Platforms without a Native implementation (e.g. macOS) continue to use RandomAccess. This cascades to all unit tests that use the default device type (Garnet server tests and the Tsavorite test projects). Also make NativeStorageDevice's misaligned-I/O guard throw TsavoriteException instead of IOException, consistent with the class's other precondition/validation guards (segment/sector-size checks); IOException remains reserved for actual kernel I/O completion failures. Update the corresponding DeviceTests assertions to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * Fix ObjectAllocator recovery flush writing past the page/segment boundary A full-page recovery flush whose fromAddress points just past the PageHeader double-counted the header: endOffset was computed as startOffset + PageSize, then startOffset was reset to 0 to include the header while keeping the inflated end, producing a PageSize + PageHeader byte write. When a segment holds a single page (SegmentSize == aligned page size), that write crosses into the next segment. Managed devices (RandomAccess) silently tolerated this by growing the segment file; NativeStorageDevice correctly rejects a write whose end exceeds the segment. Clamp the page-relative end to PageSize so a full-page flush writes exactly the page and never past it. Records never span pages, so this only removes the header double-count. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * Address code review: harden segment-boundary clamp and gate Native default to x64 Linux Two production-readiness fixes from code review: - ObjectAllocatorImpl: make the page-boundary clamp unconditional instead of only inside the isFirstRecordOnPage branch, so no flush path (mid-page start or partial snapshot, in addition to full-page recovery starting past the PageHeader) can emit a write whose end crosses the page/segment boundary. - Devices.GetDefaultDeviceType: default to Native on Linux only when the process architecture is x64, matching the shipped prebuilt native library (runtimes/linux-x64). Other Linux architectures (e.g. arm64) fall back to the managed RandomAccess device instead of failing to load the native library on the first storage IO. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * Propagate device write errors through flush/checkpoint and session teardown Two pre-existing robustness fixes surfaced by the code review of the native device switch (the native device surfaces real IO errors that managed devices previously masked). Both are error-path only; success paths are unchanged. Durability: on a device write error, the object-allocator flush completion (CountdownCallbackAndContext, reached via CircularDiskWriteBuffer) hardcoded errorCode 0 when invoking the upper-layer callback, so AllocatorBase's AsyncFlushPageCallback (which is already error-aware and records failures in errorList) treated a failed flush as success and advanced FlushedUntilAddress past unwritten data. Retain and forward the first non-zero error instead. Likewise, the snapshot, main-index, overflow-bucket, and commit-metadata checkpoint completions logged the error but signaled success; they now fault their completion (TrySetException / throw) so a checkpoint cannot commit after a failed write. Session: RespServerSession.TryConsumeMessages caught a fatal exception, disposed the sender, and returned 0, letting the network receive loop re-enter on the disposed sender (null response-object pointers -> NullReferenceException). It now rethrows; both the TLS and non-TLS receive loops already tear the connection down cleanly on exception. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * Revert RESP session rethrow; keep TryConsumeMessages as an exception firewall Reverts the earlier change that made RespServerSession.TryConsumeMessages rethrow from its catch-all. TryConsumeMessages is intentionally an exception firewall: it is invoked from the network IO-completion/receive machinery, and the broad catch guarantees no unexpected exception escapes into that machinery. On error it logs, disposes the sender (closing the connection), and returns so the server degrades gracefully and stays stable. The rethrow was not required for the native-device switch (the failing cluster tests were fixed by the ObjectAllocator recovery-flush fix), and it changes a long-standing network hot-path contract with subtle cross-path implications (TLS async readers, Lua redis.call). Any teardown-cleanliness improvement belongs in a separate, dedicated change with TLS + Lua stress coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * [Tests] Harden MigrateVectorStressAsync against transient nil results during migration The vector-set migration stress test hammers VADD/VEMB while slots migrate back and forth. Its retry loop already treats MOVED/timeout/connection as retryable migration transients, but a raw Execute can also transiently return a nil/null RedisResult mid-migration. Casting that nil to int threw a NullReferenceException that escaped the retry filter (and the VEMB read cast a nil to a null string[] that would NRE on .Length), intermittently failing the test. Switching the default Linux device to NativeStorageDevice shifts IO timing during migration and made this pre-existing flake surface more often. Treat a nil/null result as one more retryable transient in both the write and read loops. Data integrity is unaffected: only writes that return 1 are recorded, and the end-of-test VEMB validation independently verifies every recorded element survived all migrations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * Address PR review comments: ToString null-check, DeviceType doc, firstErrorCode note - CountdownCallbackAndContext.ToString() now null-checks `context` (it was guarded by the `callback` null check but dereferenced `context`, which can be null). - DeviceType.RandomAccess doc now reflects the x64 gating: the Linux native library ships only for x64, so non-x64 Linux (e.g. arm64) and macOS default to RandomAccess; Windows and x64 Linux default to Native. - Documented why firstErrorCode is intentionally not reset in CountdownCallbackAndContext.Set(): a fresh instance is created per partial flush (OnBeginPartialFlush -> new()), and a device write can RecordError before OnPartialFlushComplete installs the callback via Set(); resetting there would discard a pre-Set error and let a failed flush report success. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * [Tsavorite] Fix recovery flush page/segment overshoot at its root cause A recovery flush starts at scanFromAddress, which for the first record on a page is one PageHeader (64 bytes) past the page start. AsyncFlushPagesForRecovery left that flush marked non-partial while the object allocator hardcodes numBytesToWrite = PageSize, so the flush re-included the header and computed a PageSize + PageHeader-byte write. That write overshoots the page and, when a segment holds a single page, the segment boundary: managed devices silently grow the segment file, NativeStorageDevice rejects it. Mark the mid-page recovery flush as partial (fromAddress > page start). The existing partial path then derives the write end from untilAddress (the page end), and the first-record header-include still writes the whole page, so the write is exactly one page. Record selection is unchanged: the snapshot/hybrid-log boundary is carried separately via formerFlushedUntilAddress. Updates the recovery-flush assertion in ObjectAllocatorImpl to reflect that a recovery flush may be front-partial but always extends to the page end. Removes the now-dead partial recompute of numBytesToWrite in the WritePage path (the write uses alignedBufferSize and numBytesToWrite is not read afterwards). Also reverts the earlier defensive clamp, which treated the symptom. Validated: ClusterSRPrimaryCheckpointRetrieve 8/8; Tsavorite recovery suite 197/0; full cluster replication suite 107/0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * [Tsavorite] Ship musl native device + CI to build canonical prebuilts for all platforms Switching the Linux default to the native device surfaced that Alpine (musl) had no working native library: the shipped prebuilt is a glibc build whose DT_NEEDED is libaio.so.1t64, which musl's loader (and Alpine's libaio.so.1) cannot satisfy, so the first storage-tier/AOF write threw DllNotFoundException. The native C++ itself builds cleanly on musl; we simply were not shipping a musl binary. Changes: - Ship a linux-musl-x64 prebuilt (libnative_device.so + libnative_device_libaio.so) built against musl; it links the portable libaio.so.1 / liburing.so.2 SONAMEs. - Make the NativeStorageDevice loader RID-aware: it resolves runtimes/<rid>/native/ from OS + architecture + libc (linux-x64, linux-musl-x64, linux-arm64, linux-musl-arm64, win-x64, win-arm64, osx-*) instead of hardcoding linux-x64/win-x64. - Package prebuilts with a recursive glob over Device/runtimes/** so a new RID folder is shipped automatically without a csproj edit. - Devices.GetDefaultDeviceType now returns Native on musl x64 too (a real musl prebuilt is shipped); non-x64 and other unshipped RIDs still fall back to RandomAccess. - Add .github/workflows/native-build.yml: builds native_device via CMake for every RID (linux glibc/musl x64+arm64 via containers/QEMU; win x64+arm64 via MSVC), verifies the exported C ABI, and on manual dispatch opens a PR updating the checked-in prebuilts. macOS is intentionally excluded until a file_darwin.cc backend exists (the current C++ is hard-wired to libaio/io_uring/O_DIRECT and will not compile on macOS). - Add libs/storage/Tsavorite/cc/build-native.sh: reproducible per-platform build helper used by both the workflow and developers. - Update the io_uring init-failure diagnostic (container seccomp blocking io_uring_setup is the real cause; musl is now supported), the native README, .gitattributes (mark native libs binary), and the docker validation script (Alpine now tests the native device too). Validated: full docker image validation (default + native device persistence across all 5 images incl. Alpine musl) 67/0; native-build.sh recipe built + verified locally for linux-x64 glibc, linux-musl-x64, and linux-musl-arm64 (QEMU); Tsavorite device tests 89/0; full solution build clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * [Tsavorite] native-build CI: auto-detect VS generator + robust dumpbin (fix Windows jobs) The hosted Windows runner ships Visual Studio 18 (2026), so the hardcoded 'Visual Studio 17 2022' generator failed with 'could not find any instance of Visual Studio'. Let CMake auto-detect the installed VS generator (-A selects the target arch), drop the now-redundant Spectre-libs install step (the image already includes the Spectre-mitigated CRT and ARM64 toolset), and locate dumpbin via vswhere instead of a hardcoded 2022 path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * [Tsavorite] Make native device filesystem include portable (C++17) for Windows/modern toolsets Adding the Windows jobs to the native-build workflow surfaced that the C++ used <experimental/filesystem>, which the newest MSVC toolset (Visual Studio 2026 / 17.10+) has removed, so the win-x64 and win-arm64 builds failed with C1083 "Cannot open include file: 'experimental/filesystem'". Introduce filesystem_compat.h, which prefers C++17 <filesystem> and only falls back to <experimental/filesystem> where <filesystem> is unavailable, exposing a single `tsv_fs` alias. Replace the hard-coded std::experimental::filesystem uses in file_system_disk.h and native_device.h with it, and compile the native library as C++17 (bump CMAKE_CXX_STANDARD to 17; add /std:c++17 for MSVC and -std=c++17 for gcc/clang). No C++17-removed features are used (no std::auto_ptr etc.). Regenerated the checked-in linux-x64 and linux-musl-x64 prebuilts from the C++17 source. The win-x64/win-arm64 and arm64 Linux prebuilts are produced by the native-build workflow (run it with update_repo=true to refresh all of them). Validated: Tsavorite device tests 89/0 on the regenerated glibc binary; docker image validation (default + alpine, native + default device persistence) 28/0; glibc and musl builds compile clean under C++17 locally. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * [Tsavorite] native-build CI: locate Windows dll/pdb recursively The build succeeds but CMAKE_RUNTIME_OUTPUT_DIRECTORY is overridden, so native_device.dll is not at build/src/Release/. Find the dll/pdb recursively under build/ instead of assuming a generator-specific path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * [Tsavorite] native: don't pass /CETCOMPAT for ARM64 (LNK1246) /CETCOMPAT (CET shadow stack) is x86/x64-only; MSVC rejects it for ARM64 with LNK1246, breaking the win-arm64 native build. Guard the flag on CMAKE_GENERATOR_PLATFORM != ARM64 so win-x64 is unchanged and win-arm64 links. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * [Tsavorite] Add canonical native prebuilts for all 6 RIDs from native-build CI Replaces the locally-built linux-x64/linux-musl-x64 prebuilts and the stale (C++14) win-x64 dll with the canonical binaries produced by the native-build workflow, and adds the previously-missing linux-arm64, linux-musl-arm64, and win-arm64 prebuilts. All six were built from the C++17 source by .github/workflows/native-build.yml (run 30671785943, all jobs green) and had their exported C ABI verified in that workflow. Runtime-validated locally: linux-x64 (Tsavorite device tests 89/0) and linux-musl-x64 (Alpine native device active + storage-tier persistence). The arm64 and Windows binaries are build- and symbol-verified by CI; enabling Native by default on arm64 is deferred pending validation on real arm64 hardware (GetDefaultDeviceType keeps arm64 on the managed device; the prebuilts allow explicit --device-type Native there). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * [Tsavorite] native-build CI: publish rebuilt binaries onto the dispatched PR branch Reworks the workflow's publish step so a developer working on a PR that changes the native device can get the rebuilt binaries into their own PR: - Dispatching native-build on a feature branch with update_repo=true now commits the refreshed runtimes/<rid>/native/ binaries directly onto that branch (rebasing onto the latest tip first), so the open PR updates in place and ci.yml then tests the C# against the new native code. Dispatching on main/dev opens a review PR instead (protected branches are never pushed to directly). - Uses an optional NATIVE_BINARIES_PAT secret for the push so it can re-trigger ci.yml; falls back to GITHUB_TOKEN (push lands, ci.yml runs on the next push or a manual re-run) when the secret is absent. Documented in the workflow header and README. - Adds a per-branch concurrency group so two dispatches can't race on the push. - Tightens the build-only trigger to exclude cc/README.md so a docs-only edit does not spin up the 6-platform build; the filter remains scoped to the native sources under libs/storage/Tsavorite/cc/** (a change under cs/** — including the checked-in binaries the publish step commits — never triggers native-build, avoiding loops). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * [Tsavorite] native-build CI: explicitly ensure + verify Windows Spectre mitigation The Windows build requires the Spectre-mitigated CRT libraries (CMakeLists uses /Qspectre /guard:cf /sdl; a missing component fails the link with MSB8040, per cc/README.md). The workflow previously relied implicitly on the hosted image shipping them. Make it explicit and verified: - Add an "Ensure MSVC Spectre-mitigated CRT libraries" step that detects the lib\spectre\<arch> component (and the ARM64 VC toolset for the arm64 target) and installs it on demand, so the build no longer silently depends on image contents. - After the build, verify the produced native_device.dll actually carries the security mitigations by checking the PE load config for Control Flow Guard (/guard:cf) and the stack Security Cookie (/GS,/sdl) — both set alongside /Qspectre — failing the job if they are absent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * [Tsavorite] Remove macOS commentary from native-build workflow and device docs macOS is not a shipped native platform here; drop the macOS/darwin explanatory commentary from the native-build workflow header and the GetDefaultDeviceType doc comment to keep the comments focused on what is actually built and shipped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd * [Tsavorite] Update prebuilt native device binaries Regenerated by the Build Native Device workflow (run 30677804058) from the native sources on 'badrishc/switch-to-native-device-linux'. --------- Co-authored-by: badrishc <badrishc@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Copilot-Session: acafd58e-b5ce-420c-aec8-0e6a492d69fd | 1 个月前 | |
IRecordTriggers: per-record lifecycle callbacks for dispose, flush, evict, and disk read (#1695) * Wire storeFunctions.DisposeRecord into delete lifecycle Centralize record disposal at delete/expiration sites: - hlog.DisposeRecord(Deleted) now calls storeFunctions.DisposeRecord before ClearHeapFields in both ObjectAllocatorImpl and SpanByteAllocatorImpl, giving the application a single callback for cache-size tracking, external resource cleanup, etc. - InternalDelete: single DisposeRecord(Deleted) immediately after InPlaceDeleter (mutable) and before Seal (tombstone). - HandleRecordElision / TryTransferToFreeList: remove all DisposeRecord calls — record is already cleaned at the delete site. - InternalRMW: add DisposeRecord(Deleted) for ExpireAndStop and ExpireAndResume at their respective sites, including inside ReinitializeExpiredRecord for the IPU path. Fix pre-existing CAS-failure bug where ClearSourceValueObject disposed the source before CAS (now deferred to post-CAS success via ClearValueIfHeap). Fix ReinitializeExpiredRecord to set tombstone on the source when reinitialize-in-place fails for IPU, so CreateNewRecordRMW uses InitialUpdater instead of CopyUpdater on the disposed source. - DisposeRecordsInRangeForEviction: skip tombstoned records — they were already disposed with DisposeReason.Deleted at the delete site. - Remove all manual heap disposal (AddHeapSize, DisposeValueObject, ClearValueIfHeap) from Garnet InPlaceDeleter and RMW expiration paths in ObjectStore and UnifiedStore session functions. - GarnetRecordDisposer.DisposeRecord handles heap-size tracking only for DisposeReason.Deleted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename IRecordDisposer → IRecordTrigger Mechanical rename across the codebase to reflect the broadened responsibilities (dispose + flush callbacks). Renamed: - IRecordDisposer → IRecordTrigger - DefaultRecordDisposer → DefaultRecordTrigger - SpanByteRecordDisposer → SpanByteRecordTrigger - GarnetRecordDisposer → GarnetRecordTrigger - TRecordDisposer → TRecordTrigger - recordDisposer → recordTrigger - Test disposer types accordingly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add OnFlushRecord and OnDiskReadRecord callbacks to IRecordTrigger Generic infrastructure for per-record callbacks during page flush and disk reads. These are not BfTree-specific — any record type with external resources can use them. - Add OnFlushRecord(ref LogRecord) + CallOnFlush gate to IRecordTrigger Called on original in-memory records in OnPagesMarkedReadOnlyWorker before pages are flushed to disk. - Add OnDiskReadRecord(ref LogRecord) + CallOnDiskRead gate to IRecordTrigger Called at all 4 ClearBitsForDiskImages sites: Recovery.cs (page scan), AllocatorBase.cs (delta log + async disk read), AllocatorScan.cs (push scan). - Wire through IStoreFunctions and StoreFunctions - Add OnFlushRecordsInRange to ObjectAllocatorImpl - Default implementations: CallOnFlush/CallOnDiskRead => false - GarnetRecordTrigger: CallOnFlush/CallOnDiskRead => false (no BfTree on this branch) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename IRecordTrigger API and separate OnEvict from OnDispose Rename all API members for clarity and consistency: - IRecordTrigger → IRecordTriggers (plural: collection of callbacks) - TRecordTrigger → TRecordTriggers, recordTrigger → recordTriggers - DisposeRecord → OnDispose - DisposeValueObject → OnDisposeValueObject - OnFlushRecord → OnFlush - OnDiskReadRecord → OnDiskRead - DisposeOnPageEviction → removed (replaced by CallOnEvict) - GarnetRecordTrigger.cs → GarnetRecordTriggers.cs - IRecordTrigger.cs → IRecordTriggers.cs Separate page eviction from disposal: - Add OnEvict(ref LogRecord) + CallOnEvict as a distinct lifecycle callback - Remove DisposeReason.PageEviction — eviction is not a disposal - EvictRecordsInRange calls OnEvict instead of OnDispose(PageEviction) - OnDispose only handles true disposal reasons (Deleted, CAS failures, etc.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix: revert accidental rename of AsyncIOContext.DisposeRecord The blanket sed renamed AsyncIOContext.DisposeRecord() to OnDispose(), but this method disposes IO resources — it's unrelated to IRecordTriggers. Reverted to DisposeRecord() for AsyncIOContext and all its call sites. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Pluralize struct names to match IRecordTriggers convention GarnetRecordTrigger → GarnetRecordTriggers DefaultRecordTrigger → DefaultRecordTriggers SpanByteRecordTrigger → SpanByteRecordTriggers TrackingRecordTrigger → TrackingRecordTriggers (test) ObjTrackingRecordTrigger → ObjTrackingRecordTriggers (test) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove CacheSizeTrackerHolder; use late-bound CacheSizeTracker directly - Make GarnetRecordTriggers a readonly struct with readonly CacheSizeTracker field (reference type survives struct copy, no defensive copies with readonly struct) - Add CacheSizeTracker() parameterless ctor + Initialize(store, ...) for late-bind - Creation order: new CacheSizeTracker() → new GarnetRecordTriggers(tracker) → new TsavoriteKV(...) → tracker.Initialize(store, ...) - Remove CacheSizeTrackerHolder wrapper class entirely Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename OnFlushsInRange → FlushRecordsInRange Consistent with EvictRecordsInRange naming. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix EvictRecordsInRange doc comments to reference OnEvict Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Call storeFunctions.OnDispose for all disposal reasons Remove Deleted-only gate — OnDispose is now invoked for all DisposeReason values (Deleted, CAS failures, reviv freelist, etc.) in both ObjectAllocatorImpl and SpanByteAllocatorImpl. The application filters by reason in its implementation as needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: clean up IRecordTriggers.cs - Remove redundant 'public' modifiers from interface properties - Make DefaultRecordTriggers and SpanByteRecordTriggers readonly structs - Remove unnecessary 'unsafe' from SpanByteRecordTriggers.OnDisposeValueObject - Remove 'readonly' from struct members (redundant with readonly struct) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> | 4 个月前 | |
Initial commit | 2 年前 | |
Replace net9 with net10 in projects and CIs (#1518) * replace net9 with net10 in project files * Update net9.0 to net10.0 in ci and nightly * Updated BDN to run net10 instead of net90 the website is not done yet. * Updating BDNs to not use .net9.0. Unfortunately, .net10 is not supported in current BDNs. * Updated External Release to release 8.0 and 10.0 only (no 9.0) * fix * Enabled Win10 for BDN since recent fix * Updated a couple BDNs expected values that were a bit out of range * Updated .net9.0 for CodeQL --------- Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> | 7 个月前 | |
Implement `VSETATTR` (#2001) * sketch out vsetattr impl * fixes + tests; diskann is not correctly handling attribute removal, so test fails * add failing replication test for vsetattr * fix vsetattr replication * bump diskann-garnet to 4.0.4 * address feedback | 1 个月前 | |
[Tsavorite] Add native Linux storage backend, harden NativeStorageDevice, refresh storage benchmarks (#1831) * Port device/IO changes from optimize-v2-io onto kv-bench All 31 non-benchmark files changed on optimize-v2-io (vs its branch base d3677cfaa) ported here. Backup tag: optimize-v2-io-prerebase-backup @ 3f41f2bdf. Scope: device/IO/native-backend ONLY. Includes: Tsavorite C++ native device: - io_uring backend + pluggable C ABI (file_linux.cc/h) - error model split (native_device_error.h) - file_system_disk + native_device.h updates - CMakeLists + README Tsavorite C# device: - NativeStorageDevice: IoBackend enum (Default, Libaio, Uring), completion threads, production-readiness pass - LinuxFileExtensions.cs: P/Invoke open() for true O_DIRECT - ManagedLocalStorageDevice + RandomAccessLocalStorageDevice: O_DIRECT wiring on Linux - Devices.cs: router updates for new device APIs Tsavorite allocator + utilities: - AllocatorBase: bounded backoff in TryAllocateRetryNow - CompletionEvent: Wait(TimeSpan) overload Tsavorite checkpoint management: - LocalStorageNamedDeviceFactory + Creator surface ioBackend + completionThreads parameters Tests: - DeviceTests.cs updated for new device APIs Garnet host: - --device-io-backend, --device-completion-threads flags - defaults.conf updated, GarnetServerOptions wiring Dockerfiles (all 5): install liburing alongside libaio. Note: YCSB.benchmark and KV.benchmark are not modified by this commit; KV.benchmark is the supported benchmark on this branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * KV.benchmark: add uring backend, fix --validate for disk-spill, doc liburing runtime dep - Options: --device-io-backend now accepts 'uring' (aliases: io_uring, iouring) in addition to libaio/default; help text + validation error list updated to match. - KV.benchmark README: device-backend table now describes the libaio vs uring split, with a runtime-install snippet for liburing across Debian/Ubuntu, Fedora/RHEL/AzureLinux, and Alpine, plus link to the full Tsavorite Native Device docs. New 'native + libaio' and 'native + uring' rows added to the cookbook for constrained-log large-dataset workloads. - Tsavorite Native Device README: new top-level 'Runtime dependencies (end users)' section listing the apt/dnf/apk install lines and how to fall back to the no-liburing variant. - KV.benchmark Validate: fix two bugs that surface when load and run use different thread counts and when the log spills to disk: 1) writerThread reconstruction now uses ResolvedLoadThreads (not Options.Threads which is the RUN count), so --load-threads N with --threads M != N validates correctly. 2) Reads of records below HeadAddress return Status.IsPending; the previous code counted these as misses. Validate now issues reads in batches of 256 and drains via CompletePendingWithOutputs, verifying each completed output against the per-thread pattern. Verified end-to-end with both backends: 4.6M × 100B × 8T × log=256m (~580MB dataset > 256MB log → forces disk spill) × 50R/50U × --validate: native + libaio → [validate] OK, run = 1.27 M ops/s native + uring → [validate] OK, run = 1.02 M ops/s 4.6M × 100B × 8T × log auto (fits) × 95R/5U × --validate: native + libaio → [validate] OK, run = 16.10 M ops/s native + uring → [validate] OK, run = 16.33 M ops/s Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite: replace MarkHandleAsAsync reflection hack with RandomAccess on SafeFileHandle Background: MarkHandleAsAsync used reflection to flip SafeFileHandle.IsAsync's non-public setter so a P/Invoke-opened O_DIRECT FD could be wrapped in 'new FileStream(handle, isAsync: true)' without throwing 'Handle does not support asynchronous operations'. The flag is non-public in .NET 8/10, the hack was fragile across future runtime versions, and on Linux IsAsync is a contract gate (no real overlapped I/O exists for files), so the lie bought us nothing beyond letting the FileStream constructor accept the handle. RandomAccessLocalStorageDevice — refactor: - StorageAccessContext.handle is now SafeFileHandle (was FileStream). - CreateRead/WriteHandle: * Linux + O_DIRECT capable: LinuxFileExtensions.OpenDirect -> raw SafeFileHandle, no FileStream wrap. Page-cache bypass via the O_DIRECT flag at open(2), exactly as before. * Otherwise (Windows; or Linux when filesystem rejects O_DIRECT): File.OpenHandle(path, ..., FileOptions.Asynchronous | cast FILE_FLAG_NO_BUFFERING). On Windows this gives the runtime IOCP-bound OVERLAPPED I/O; on Linux it's page-cached. - All I/O goes through RandomAccess.{Read,Write}Async(safeHandle, memory, offset). On Windows: true kernel async via IOCP. On Linux: pread/pwrite dispatched to ThreadPool (same as before). - GetFileSize uses RandomAccess.GetLength(handle). - SetFileSize uses RandomAccess.SetLength(handle, size). LinuxFileExtensions: - MarkHandleAsAsync and the IsAsyncProperty reflection are deleted entirely (no remaining callers). - System.Reflection using removed. ManagedLocalStorageDevice: - Reverted to origin/main. This device is designed to stay within FileStream APIs; the O_DIRECT branch we added doesn't belong here. Verified: - Tsavorite test.hlog DeviceTests: 36/36 passed. - KV.benchmark --device randomaccess --log-memory 256m --preallocate-log --rumd 50,50,0,0 --validate: PASS, 357 K ops/sec, iostat shows 100-177 K real disk r/s and 53-90% NVMe util → O_DIRECT page-cache bypass confirmed. - KV.benchmark --device randomaccess (log fits) --validate: PASS, 15.7 M ops/sec. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite tests: cross-device hardening suite (Native + RandomAccess + ManagedLocal) The Phase-7 hardening suite added in this branch was NativeStorageDevice- specific. Add parametrized variants of the four tests that are pure IDevice contract checks (not native-specific lifecycle/API), so they exercise all three local-storage device implementations: - Hardening_AllDevices_RoundTrip_BasicReadWrite - Hardening_AllDevices_RoundTrip_AcrossSegmentBoundary - Hardening_AllDevices_Parallel_32ConcurrentWrites - Hardening_AllDevices_Parallel_BurstyTraffic Each is parametrized by a new DeviceKind enum (Native, RandomAccess, ManagedLocal). Native is gated on OperatingSystem.IsLinux() (the C++ shim links against libaio/liburing); the other two run on both Linux and Windows. A shared CreateDeviceForTest helper takes care of the per-kind ctor + Initialize() dance so the test body stays uniform. Result: 38/38 hardening tests pass on Linux (12 new cross-device + 26 native-only). Native-specific tests retained as-is because they test API that doesn't exist on the other devices: - Lifecycle (DisposeBeforeInitialize, InitializeTwice, etc.) — NativeStorageDevice defers Initialize from the ctor; the other devices initialize in their ctor. - Segment-size validation (NonPowerOfTwoSegmentSize_Throws, etc.) — Initialize() is the only callsite that validates. - Recovery_*_SegmentSize_* — the native device's open() path is the only place that records and re-validates per-segment-size metadata. - SectorSize stability across opens — not all devices expose this. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite device tests: split into IDevice_ contract + NativeStorageDevice_ buckets; fix AsyncPool creator-throw hang Test refactor: - IDevice_*: 8 contract tests parametrized across Native, RandomAccess, ManagedLocal (round-trip basic, round-trip cross-segment, round-trip various segment sizes, 32 concurrent writes, 64 concurrent reads, mixed reads+writes, bursty traffic, stress burst of 100 writes, permission-denied callback contract). 33 cases total. - NativeStorageDevice_*: 16 native-only tests for behaviors managed devices don't have (deferred Initialize signature, recovery segment-size mismatch detection, sector-size discovery, sync-throw unaligned IO guard). AsyncPool fix: GetOrAdd reserved a slot in totalAllocated before calling creator(). If creator() threw (e.g. open() returned EACCES, ENOSPC), the slot was never released, so Dispose() would loop forever waiting for totalAllocated to drain to zero. This manifested as a process hang when a device pool's first open() failed. Rollback the reservation on exception so the failure propagates cleanly and the pool can still be disposed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite IDevice: unify Initialize contract — required for all devices, -1 = unbounded single segment Before this change, IDevice.Initialize had the same signature for every device but very different semantics: * StorageDeviceBase (RandomAccess, ManagedLocal, LocalStorage, NullDevice, LocalMemoryDevice): the ctor pre-set segmentSize = -1 / bits = 64 / mask = ~0, so calling an IO entry point without Initialize() silently ran in unbounded single-segment mode. * NativeStorageDevice: Initialize was MANDATORY (the C++ shim needs the segment size at create time for libaio/io_uring geometry), IO entry points threw if invoked first, and segmentSize = -1 was rejected. This commit unifies the contract: every IDevice must call Initialize() exactly once before any IO, and segmentSize = -1 selects unbounded single-segment mode on every device. Implementation: * StorageDeviceBase - Added `initialized` flag (volatile) and `EnsureInitialized()` helper that throws InvalidOperationException with a clear message naming the device by FileName. - Ctor leaves `initialized = false` but keeps the safe fallback defaults (-1 / 64 / ~0) so any cold maintenance path that touches segmentSizeBits before the guard can't compute outright nonsense. - EnsureInitialized() called from the base address-based ReadAsync / WriteAsync overloads and TruncateUntilAddress / TruncateUntilAddressAsync. - Initialize sets `initialized = true` at the end. * NativeStorageDevice - Accepts segmentSize = -1: translates to 1UL << 63 for the native shim so the C++ FileSystemSegmentedFile's shift = log2(segment_size) math collapses every non-negative upper-layer address into segment 0 (parity with the managed-side bits = 64 / mask = ~0). Single growing file on disk. - Tracks the value passed to native in nativeSegmentSizeBytes (replaces the diagnostic-only configuredSegmentSizeBytes long field, which couldn't hold 1<<63 without overflow). - ABI readback (NativeDevice_GetSegmentSize) compared against the value we sent to native, not the user-facing -1. - Always rejects omitSegmentIdFromFilename — the C++ shim has no omit-suffix code path, every segment is written as <base>.<segmentId>. Better to fail fast than silently produce wrong file names. * Concrete IO entry points (ReadAsync / WriteAsync / RemoveSegment / RemoveSegmentAsync) of NullDevice, LocalMemoryDevice, ManagedLocalStorageDevice, RandomAccessLocalStorageDevice, LocalStorageDevice, AzureStorageDevice, ShardedStorageDevice, and TieredStorageDevice now call EnsureInitialized() before doing work. Caller fix-ups: * LocalStorageNamedDeviceFactory.Get now calls device.Initialize(-1L) before returning. Commit / checkpoint metadata is single growing-file usage (segment 0 only, .0 suffix), so unbounded mode is the right default and unblocks every DeviceLogCommitCheckpointManager caller from needing to remember to initialize. * LocalStorageNamedDeviceFactory.ListContents skips dotfile entries — defensive against a pre-existing race in LinuxFileExtensions.IsDirectIOSupported where a .tsavorite-odirect-probe-* temp file can leak in the commit dir if File.Delete races with File.GetFiles. Without this filter, leaked probe files surface as Int64.Parse("") failures in DefaultCheckpointNamingScheme.CommitNumber. * SimulatedFlakyDevice.Initialize now propagates to the wrapped device. * ComponentRecoveryTests Setup_* helpers call Initialize(-1) on devices they construct directly (bypass the Tsavorite allocator path which normally Initializes). Tests (DeviceTests.cs): * IDevice_ReadAsyncBeforeInitialize_Throws(kind) × 3 — new contract test. * IDevice_WriteAsyncBeforeInitialize_Throws(kind) × 3 — same. * IDevice_Initialize_SegmentSizeMinusOne_UnboundedSingleSegment(kind) × 3 — write at offset 1 MiB (would be in segment-N for any positive size) and read back, confirming -1 routes through segment 0 on all 3 kinds. * NativeStorageDevice_Initialize_OmitSegmentIdFromFilename_Throws — new native-only test for the omit rejection in both -1 and explicit-size modes. * Removed NativeStorageDevice_{Read,Write}AsyncBeforeInitialize_Throws (now subsumed by the IDevice_ variants). Docs: * IDevice.Initialize docstring rewritten to spell out the new contract and the -1 semantics. NativeStorageDevice.Initialize remarks updated. Verified on Linux net10.0 Release: * 599 hlog tests (491 passed + 108 skipped) * 305 recovery tests * 144 + 155 + 127 + 346 = 772 other Tsavorite + Garnet RespTests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite: native devices honor omitSegmentIdFromFilename; O_TMPFILE probe (no race) Two related fixes on top of the unified Initialize contract: 1) NativeStorageDevice now supports omitSegmentIdFromFilename ───────────────────────────────────────────────────────── Previously Native rejected the omit flag because the C++ shim hard-coded the '.<segmentId>' suffix in three places in file_system_disk.h. This made the IDevice contract asymmetric (managed devices honored omit, Native didn't). Fix by threading the bool through the entire C++/C ABI: C++ (libs/storage/Tsavorite/cc/src/device/): * FileSystemSegmentBundle: new bool omit_segment_id_; both ctors accept it and use a new segment_path(idx) helper that returns just filename_ when set, otherwise filename_ + '.' + std::to_string(idx). Used at all three locations that previously hard-coded the suffix. * FileSystemSegmentedFile: new bool omit_segment_id_ (const) wired through ctor and propagated to bundles allocated by OpenSegment. * NativeDeviceImpl: new bool omit_segment_id constructor param; recorded as omit_segment_id_ member. ValidateRecoveredSegments short-circuits in omit mode (single bare-named file, segment-size mismatch check is meaningless when there's no .<id> suffix to scan for). * native_device_wrapper.cc / NativeDevice_CreateWithBackend: new trailing 'bool omit_segment_id' parameter. ABI BUMP — managed wrapper updated to match; Linux .so rebuilt and committed at libs/storage/Tsavorite/cs/src/core/Device/runtimes/linux-x64/native/ libnative_device.so. **Windows DLL must be rebuilt by user** with cmake -G 'Visual Studio 17 2022' -A x64 -T v143,spectre=true. C# (libs/storage/Tsavorite/cs/src/core/Device/): * NativeStorageDevice P/Invoke signature updated. * NativeStorageDevice.Initialize removes the 'always rejects omit' guard. It now accepts omit:true together with segmentSize = -1 and forwards to native; rejects omit:true together with a positive segmentSize with a clear error message (multiple segments would collapse onto the same on-disk path and clobber each other). Tests (DeviceTests.cs): * IDevice_Initialize_OmitSegmentIdFromFilename_BareFileName(kind) × 3: writes via Initialize(-1, omit:true) and asserts the on-disk file is the bare basename (no .0 suffix). Replaces the native-only 'throws' test from the previous commit. * IDevice_Initialize_OmitSegmentIdFromFilename_WithoutMinusOne_Throws(kind) × 3: enforces the no-positive-size-with-omit invariant on every kind. 2) IsDirectIOSupported uses O_TMPFILE (race-free probe) ───────────────────────────────────────────────────── The previous probe in libs/storage/Tsavorite/cs/src/core/Device/ LinuxFileExtensions.cs created a hidden '.tsavorite-odirect-probe-<pid>- <guid>' file in the device's directory, then File.Delete'd it in a silent-catch finally. Multiple concurrent commits (one device per Get()) ran probes simultaneously; concurrent ListContents calls from CommitRecordBoundedGrowthTest would observe the probe file during its brief lifetime, and DefaultCheckpointNamingScheme.CommitNumber would then throw FormatException on long.Parse(''). 18/20 baseline failure rate. Switching from create+unlink to open(directory, O_TMPFILE | O_RDWR | O_DIRECT) tells the kernel to allocate an anonymous inode in the directory's filesystem with NO directory entry. The probe inode is invisible to readdir/getdents regardless of timing; concurrent ListContents cannot observe it. Freed on close. Linux >= 3.11 + ext4/xfs/tmpfs/btrfs all support it. If O_TMPFILE itself fails (EOPNOTSUPP on some filesystem) we conservatively report 'no O_DIRECT' so the device falls back to the page-cache path — no named-file fallback because that's the bug we're fixing. Reverts the dotfile filter in LocalStorageNamedDeviceFactory.ListContents added by the previous commit; the underlying race is now eliminated at the kernel level so the workaround is unnecessary. 20/20 LogFastCommitTests runs pass after the change (was 2/20 on baseline, 3/20 on the previous unlink-after-open attempt which still raced). Verified on Linux net10.0 Release: * 612 hlog tests (504 passed + 108 skipped) * 305 recovery tests * 62 device tests (IDevice contract + NativeStorageDevice-specific) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * KV.benchmark: add disk-IO thread-scale sweep recipe to cookbook Captures the 100M-key disk-bound thread-scale experiment from the optimize-device branch sweep so it can be reproduced verbatim in the future: - 100M × 100B records (12.8 GB on disk) - 16 MB log so ~0.125 % of dataset is in memory and almost every read is a 4 KB random disk fetch - 8 load threads, run-threads sweep 1,2,4,8,16,32 at 15s each - One row per backend (RandomAccess / native+libaio / native+uring) Added a short note after the table explaining what to compare against (the disk's fio ceiling at 4K-aligned QD=64-per-job), the expected ~2 min wall-clock per device, and the observed per-backend plateau characteristics so the next operator knows what 'good' looks like. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Device.benchmark + KV.benchmark: robustness & flag improvements Device.benchmark fixes (previously reported throughput could be 2x inflated): - Throughput counter now tallies successful completions only. Before, every ReadAsync call was counted as success even when the kernel returned EAGAIN (Status::IOError=4 — flooded libaio io_context ring). Under --throttle-limit 0 with high QD, ~40% of "ops" were errored requests. - Per-error-code histogram printed at end of run; no per-error Console.WriteLine (was emitting millions of serial writes/run, both falsifying numbers and slowing the real path). - DEBUG data validation skipped on errored ops (was reading garbage from the destination buffer on EAGAIN paths and reporting spurious "Data mismatch"). - --throttle-limit help text documents the libaio kernel ring trap (128 slots wide; high QD + no throttle floods it) and recommends --throttle-limit 128 (also the io_uring SQ depth this build uses). - --io-backend flag added (libaio / uring / default) so the existing Linux Native path can be exercised against either backend. Unknown values are rejected with an actionable message at startup instead of silently falling back to default. - --completion-threads is now wired through to the Linux Native ctor (was hardcoded to 1). - --file-size widened to long (was int; --file-size > 2GB threw a parse error). KV.benchmark + Devices.cs: clarified XML/help text for the existing --device-completion-threads / numCompletionThreads parameter to describe the current behavior (multiple drainer threads share one kernel io_context / io_uring per device). No behavior change for KV.benchmark or core Tsavorite Devices.cs API beyond the Device.benchmark surface and clarified docs. * Tsavorite Native: shard io_uring per completion thread; new C ABI Adds N independent kernel io_contexts (libaio) / io_urings (uring) per NativeStorageDevice, with completion threads bound 1:1 to contexts. libaio internally always uses 1 context (sharding empirically gave nothing — kernel mutex efficient at all tested loads), but the sharded ABI surface is kept across both backends so the templated NativeDeviceImpl<HandlerT> doesn't need a fork. C ABI changes (libnative_device.so): - NativeDevice_CreateWithBackend signature bumped: trailing int32 num_io_contexts. - New exports: NativeDevice_QueueRunFor(device, ctx_idx, timeout_secs), NativeDevice_NumIoContexts(device). - Legacy NativeDevice_QueueRun kept; under uring sharding it scans all rings (back-compat for any single-thread drainer). C# (NativeStorageDevice): - Synchronous ABI probe at Initialize() that converts EntryPointNotFoundException into a clear TsavoriteException listing the missing exports and how to rebuild — guards against a stale .so silently hanging Dispose's drain loop. - Probe is intentionally gated by the QueueRun branch so it runs only on backends that actually use the new symbols at runtime: Linux Native (libaio / uring) where QueueRun returns >= 0, NOT on Windows IOCP where the ThreadPoolIoHandler returns -1 by design. This means a stale Windows DLL keeps working unchanged because it never calls the sharded exports. cdecl/x64 ABI silently tolerates the new trailing num_io_contexts arg on NativeDevice_CreateWithBackend. - Completion threads bound 1:1 via QueueRunFor(ctxIdx) (no closure capture bug — ctxIdx is captured per-iteration into a local). - numCompletionThreads is the user-facing knob; native side decides how many contexts to actually create (libaio: always 1; uring: honours the request). Empirical justification (Device.benchmark, NVMe, 4K random reads, batch=4096, throt=512, fio ceiling 749K, NUMA0-pinned): libaio CT=1 (always 1 ctx + 1 drainer): 755K ops/sec (t=32) uring CT=1 (1 ring + 1 drainer): 357K ops/sec ← SpinLock-bound uring CT=1 ring + N drainers (regresses): 357K → 274K ← cq_lock contention uring CT=4 (4 rings + 4 drainers): 745K ops/sec uring CT=8 (8 rings + 8 drainers): 758K ops/sec ← hardware ceiling Both backends now reach the hardware NVMe ceiling. uring requires sharding (the user-space SpinLock around io_uring_get_sqe + prep + submit is the real cap). libaio doesn't need sharding (kernel io_context mutex already efficient at all tested loads). Files: - file_linux.h : UringIoHandler sharded (vector<io_uring*>, per-ring sq_lock + cq_lock, atomic round-robin pick_ring). QueueIoHandler unchanged on the data plane (single io_context_t) but exposes the same num_contexts()/TryCompleteFor/QueueRunFor surface as inline stubs for ABI symmetry. - file_linux.cc : new UringIoHandler impls; QueueIoHandler unchanged. - file_windows.h: stub overloads (num_contexts()=1, QueueRunFor=-1, 2-arg ctor) so the templated NativeDeviceImpl compiles unchanged. - native_device.h, native_device_wrapper.cc: ABI plumbing as above. - NativeStorageDevice.cs: ABI probe + per-context drain workers. - runtimes/linux-x64/native/libnative_device.so: rebuilt with sharding. * Device.benchmark: add cookbook README showing how to saturate ~750K NVMe IOPS Captures the verified copy-paste recipe for both Linux Native backends (libaio and io_uring) to hit the hardware ceiling on a Dell P5600-class NVMe, alongside a flag reference, output-schema explanation, and troubleshooting table. Headline recipes verified end-to-end on the reference setup: libaio --completion-threads 1 --threads 16 --throttle-limit 512 → 743K ops/sec uring --completion-threads 8 --threads 16 --throttle-limit 512 → 738K ops/sec (Both within 2 % of the table values in the README; zero kernel-side errors.) Key facts documented: - libaio always uses one io_context in this build regardless of --completion-threads (sharding empirically gave nothing; the hint is ignored). Pass 1 explicitly so scripts are self-describing. - io_uring needs sharded rings (CT >= 4) to escape the per-ring user-space SpinLock cap around io_uring_get_sqe + prep + submit; CT=8 is the safe peak. - --throttle-limit must be set to at least the per-ring/per-context depth (128 in this build for both backends). --throttle-limit 0 floods the kernel ring and the benchmark correctly surfaces Status::IOError=4 in the per-code histogram rather than tallying errored ops as throughput. - --file-size must be a multiple of 1024 × --sector-size (fill phase uses a 1024-sector temp buffer). Plus a section comparing Device.benchmark vs KV.benchmark to direct readers to the right tool: Device.benchmark for IO-layer ceiling validation (saturates NVMe), KV.benchmark for full-path throughput (currently caps ~30 % below the IO ceiling on the upper-layer pending-read path — see KV.benchmark README for that side of the story). Also adds a one-paragraph pointer in benchmark/README.md so the new README is discoverable from the top-level benchmarks listing. * Tsavorite Native: bounded sched_yield retry on transient kernel-ring full ScheduleOperation in both QueueFile (libaio) and UringFile (io_uring) now retries the kernel-side submission on the transient back-pressure signal (libaio: `io_submit == 0`; uring: `io_uring_get_sqe == nullptr`) up to kMaxSubmitRetries = 8 attempts, each separated by a sched_yield(). Permanent errors (libaio io_submit < 0, uring io_uring_submit < 0) are NEVER retried — they surface immediately as Status::IOError. Motivation: the upper-layer throttle gate in AllocatorBase.AsyncGetFromDisk is a racy test-then-increment (Throttle() reads numPending non-atomically, then ReadAsync does Interlocked.Increment). With N concurrent submitters all passing the gate at numPending == ThrottleLimit, in-flight can spike to ThrottleLimit + N momentarily, exceeding the 128-slot per-context / per-ring kernel ring depth when N > 8 (which is normal for Garnet under heavy disk-bound load). Pre-fix, the kernel rejects the overshoot submissions with EAGAIN, which Tsavorite handled by re-routing through the full AllocatorBase pending-read retry loop (correct but expensive: a full round-trip per IOError). Post-fix, the burst is absorbed locally by a handful of sched_yields and never surfaces to the upper layer. Why sched_yield + bounded retries is the right shape: on a 750K-IOPS NVMe the kernel ring drains a slot every ~1.3 µs and sched_yield is typically 1-10 µs on Linux, so 8 retries (worst-case ~40-80 µs window) is more than enough to absorb the typical 24-slot overshoot from a 32-thread burst. For genuine sustained overload (application submission rate exceeds device IOPS for seconds), the retries exhaust and Status::IOError surfaces — which is the correct signal for the caller to apply back-pressure. Implementation notes: - libaio: simple loop around io_submit. No lock to release; io_submit is kernel-thread-safe per io_context, so concurrent submitters serialise inside the kernel. - uring: must release sq_lock around sched_yield. Holding a SpinLock across a syscall would stall every other submitter on the same ring. Only the get_sqe == nullptr path is retried; if get_sqe succeeded we've already "consumed" an SQE slot in the user-side bookkeeping and re-issuing via get_sqe+prep on retry would corrupt the ring (we'd hold two SQEs for one logical op). For SQPOLL-disabled rings (our setup) io_uring_submit returns 1 in steady state — a non-1 there is an unrecoverable kernel-side error and surfaces immediately. Verified end-to-end (Device.benchmark, NVMe, 4K random reads, batch=4096, NUMA0-pinned, --throttle-limit 120 to match production NativeStorageDevice default): libaio CT=1: t=8 / 16 / 32 → 626K / 631K / 627K ok/sec, 0 err uring CT=8: t=8 / 16 / 32 → 622K / 625K / 626K ok/sec, 0 err At extreme intentional-overload settings (--throttle-limit 4096, t=32) errors still appear — confirming the retry budget correctly exhausts when the application is genuinely outpacing the device: libaio CT=1, t=32, throt=4096: 754K ok, 14.7M code4 err (~5% of submits) uring CT=8, t=32, throt=4096: 745K ok, 0 err (uring still produces 0 err under same overload because 8 rings × 128 = 1024 SQ slots is large enough that even gross over-submission fits within the retry budget per ring.) * Tsavorite Native: PR review fixes (3-model code review pass) BLOCKER fixes: - NativeStorageDevice.Dispose UAF race. Previously NativeDevice_Destroy(nativeDevice) ran before nativeDevice was nulled, so a concurrent guard-bypassed P/Invoke could observe a non-zero handle that points to freed memory. Fix: Interlocked.Exchange atomically captures-and-nulls the handle; destroy runs on the captured pointer. EnsureReadyOrSilent now checks disposedFlag first. HIGH fixes: - UringFile::ScheduleOperation SQE leak on transient io_uring_submit failure. After a successful get_sqe + prep, the SQE is committed to the user-side SQ ring; a -EAGAIN/-EBUSY return from io_uring_submit left the slot permanently occupied with no kernel iocb, eventually starving get_sqe forever. Fix: retry io_uring_submit (without re-preparing) up to kMaxSubmitRetries on transient negatives, with sched_yield (and sq_lock released) between attempts. - UringIoHandler::Init partial-init leak. If new SpinLock() threw after io_uring_queue_init succeeded for ring i, the already-initialized ring leaked (the class dtor doesn't run on partial construction). Fix: use std::unique_ptr RAII holders during construction; release into the member vectors only after all allocations succeed. POLISH fixes: - NativeStorageDevice.Initialize tail-throw cleanup. If base.Initialize threw after the native device and completion threads were created, both leaked. Now wrapped in try/catch that cancels token, joins threads, and destroys the native device. - UringIoHandler rule-of-5 hygiene: explicitly deleted copy ctor, copy-assign, and move-assign so the implicit shallow copies (which would double-delete the raw owning pointers) cannot be generated. - DispatchUringCqe: added static_assert(is_trivially_destructible<IoCallbackContext>) so a future non-trivial member fails the build instead of silently leaking. - NativeDeviceImpl::num_io_contexts: removed unnecessary const_cast (the underlying num_contexts() is already const on both backends). - Removed duplicate XML <summary> on NativeStorageDevice.Dispose. - FileSystemDisk dead-code ctor: passed the now-required 5th arg to FileSystemSegmentedFile so the file compiles if anyone instantiates it. Comment hygiene sweep (per explicit review-rule #4 "comments should not refer to design thought processes"): - Removed embedded benchmark results, hardware-specific throughput numbers, and historical narrative from class/method documentation in file_linux.{h,cc}, native_device.h, native_device_wrapper.cc, NativeStorageDevice.cs. - Kept WHAT each method does and the invariants it enforces; moved WHY this approach was chosen out of source comments (the commit log is the appropriate place for that context). - Net: -162 lines across 7 files, no behavior change from the trim itself. Verified post-fix performance unchanged (Device.benchmark, NVMe, 4K random reads, batch=4096, NUMA0-pinned): libaio CT=1 t=16 throt=512: 746K ok/sec, 0 err (hardware ceiling) uring CT=8 t=16 throt=512: 743K ok/sec, 0 err (hardware ceiling) libaio CT=1 t=32 throt=120: 636K ok/sec, 0 err (production default) uring CT=8 t=32 throt=120: 618K ok/sec, 0 err (production default) * Tsavorite Native: make Initialize idempotent via lazy native-handle creation NativeStorageDevice.Initialize used to perform all the heavy work (native device creation, completion-thread spawn, ABI / segment-size / sector-size cross-checks) eagerly and threw "called more than once" on a second call. The other IDevice implementations (LocalStorageDevice, RandomAccessLocalStorageDevice, ManagedLocalStorageDevice) all inherit a metadata-only StorageDeviceBase.Initialize that simply overwrites segmentSize / segmentSizeBits / mask fields and is silently idempotent. They open their per-segment OS handles lazily inside the IO methods. This contract mismatch broke any caller that invokes Initialize twice on the same NSD instance. The canonical case is LocalStorageNamedDeviceFactory.Get(), which calls Initialize(-1L) as a defensive pre-init so consumers can't forget; the consumer (snapshot checkpoint state machine SnapshotCheckpointSMTask, cluster checkpoint streaming TsavoriteCheckpointReader.CreateCheckpointDevice) then calls Initialize(actualSegmentSize). Under the old NSD that throws; under the new NSD it works the same way the other backends do. Implementation: - NSD.Initialize is now metadata-only — delegates to base.Initialize. Pre-flight argument validation (segmentSize power-of-two, sector-size floor, omitSegmentIdFromFilename) is preserved. - New EnsureNativeDeviceCreated() does the heavy work, lazily, on first IO. Reads the latest base.segmentSize and base.OmitSegmentIdFromFileName so whichever Initialize call ran most recently wins. - Thread-safe via double-checked locking on a new nativeCreateLock. The publish of nativeDevice uses Volatile.Write so a second observer of nativeDevice != IntPtr.Zero is guaranteed to see a fully-initialised handle with completion threads already running. - Dispose now also takes nativeCreateLock around the cancel-join-destroy sequence so it cannot race with a concurrent EnsureNativeDeviceCreated (which would otherwise leak a freshly-published native handle and its completion threads). - IO entry points (ReadAsync, WriteAsync) call EnsureNativeDeviceCreated() before submission. Bookkeeping entry points (Reset, TryComplete, GetFileSize, RemoveSegment) no-op when the native handle has not been created yet, matching the semantics of the other backends (Reset on a device with no open handles is a no-op). Verified against the full unit-test sweep with Native forced as the default device (the GetDefaultDeviceType hack is local-only and not in this commit): Tsavorite.test: 206 / 206 (was 204 / 206 pre-fix) Garnet.test: 789 / 789 (was 110 / 792 pre-fix; 681 were blocked on Initialize-twice) Garnet.test.acl: 425 / 425 Garnet.test.collections: 746 / 746 Garnet.test.complexstring: 386 / 386 Garnet.test.rangeindex: 62 / 62 Garnet.test.vectorset: 42 / 42 No change to behavior for callers that invoke Initialize once with the real segment size, which is what every production code path already does. * Tsavorite Native: probe GetFileSize/RemoveSegment without forcing native handle GetFileSize and RemoveSegment must report the on-disk state regardless of whether IO has flowed through the device, matching LocalStorageDevice and RandomAccessLocalStorageDevice semantics. Before this fix, both no-op'd when no native handle had been created — which silently truncated the cluster manager's recovery decision because ClusterManager.cs:79 and ReplicationManager.cs:160 call `device.GetFileSize(0) > 0` to decide whether to recover persisted cluster config / replication history. With Native, a restarted node would always "Initialize new node instance config" instead of recovering, get a fresh node ID, and fail every replication-resume test (e.g. ClusterSRNoCheckpointRestartSecondary which restarts a replica and then waits for AOF sync to catch up). Changes: * GetFileSize now falls back to FileInfo when no native handle exists — same shape as RandomAccessLocalStorageDevice.GetFileSize (open-on-demand) but without paying io_uring/libaio setup cost just to stat a file. * RemoveSegment now falls back to File.Delete when no native handle exists — same shape as LocalStorageDevice / RandomAccessLocalStorageDevice (best-effort unlink, swallows ENOENT). * Per IDevice contract enforced in 889def4 ("Tsavorite IDevice: unify Initialize contract — required for all devices, -1 = unbounded single segment"), ReadAsync / WriteAsync now call EnsureInitialized() before EnsureNativeDeviceCreated() so the IDevice_*BeforeInitialize_Throws hardening tests get the same InvalidOperationException shape from Native that they get from the other devices. * Two device tests updated to match the lazy-Initialize contract that was introduced in commit f4e3044 ("Tsavorite Native: make Initialize idempotent via lazy native-handle creation"): - NativeStorageDevice_InitializeTwice_Throws → _Idempotent: idempotent Initialize matches the LSD/RA contract used by LocalStorageNamedDeviceFactory.Get + consumer re-init pattern. - NativeStorageDevice_Recovery_LargerExistingSegment_DetectsMismatch: the C++ ValidateRecoveredSegments check now fires on first IO (when EnsureNativeDeviceCreated runs), not at Initialize time, so the test asserts on a ReadAsync rather than Initialize. Verified on Linux x64 / .NET 10: * libs/storage/Tsavorite/cs/test/test.hlog: all IDevice + NativeStorageDevice tests pass (62/62) with Native default * test/cluster/Garnet.test.cluster.replication: all 4 ClusterSRNoCheckpointRestartSecondary variants pass with Native default (regression-test for the recovery path) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: wake completion drainer on Dispose via no-op IO Without this fix, NSD.Dispose() can stall up to CompletionWorkerTimeoutSecs (1s) per io_context because the completion-drainer thread is blocked in io_getevents / io_uring_wait_cqe_timeout waiting for events that will never come (the IO drain phase has already brought numPending to 0). The Thread.Join following completionThreadToken.Cancel() then has to wait for the next QueueRunFor timeout to fire so the thread can observe cancellation and exit. This was visible as exactly-1.0s gaps in cluster replication recovery traces: checkpoint metadata reads / writes that each create+dispose a fresh NSD spent ~1s in Dispose, multiplying across the ~5–10 devices created per checkpoint into multi-second stalls. ClusterReplicaSyncTimeoutTest (replicaSyncTimeout=1s) and MultiDatabaseSaveRecoverByDbIdTest(True) (2s LASTSAVE poll window) failed because of this; the actual I/O on Native is microseconds, not seconds. Fix: post a synthetic wake-up event on each io_context when Dispose runs. * libaio: submit a 0-byte read on a /dev/null fd opened in the handler ctor. /dev/null completes immediately and does not require O_DIRECT alignment, so the wake-up does not interfere with the real segment files. * io_uring: submit io_uring_prep_nop with user_data = nullptr; the drain loop recognises nullptr as a wake-up sentinel and skips dispatch. * Windows ThreadPoolIoHandler has no dedicated drainer (callbacks fire on threadpool threads), so its Wake is a no-op stub returning 0. The completion thread wakes from its blocking syscall almost immediately, observes the cancellation token on its next loop iteration, and exits. No extra idle work, no polling, no shortened timeout. * NSD.Dispose latency: 1025ms worst case -> ~20-30ms (microbenchmark). * ClusterReplicaSyncTimeoutTest with Native: ~22-25s (fail) -> ~3s (pass). * MultiDatabaseSaveRecoverByDbIdTest(True) with Native: timeout (fail) -> ~6s (pass). * Idle drainer syscall rate is unchanged (1/s/context). C ABI changes (additive — old exports preserved): * NativeDevice_WakeCompletionWorker(device, ctx_idx). * INativeDevice::Wake; QueueIoHandler::Wake, UringIoHandler::Wake, ThreadPoolIoHandler::Wake stub. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite IDevice: make Initialize() optional — ctor defaults are valid for IO Background: commit 889def4 ("unify Initialize contract — required for all devices") added an EnsureInitialized() guard that threw InvalidOperationException at every IO entry point if Initialize() had not been called first. This was redundant: the ctor already establishes segmentSize=-1 / segmentSizeBits=64 / segmentSizeMask=~0UL, which is functionally identical to having called Initialize(-1) — every absolute address right-shifts to segment 0, producing unbounded single-segment routing. The mandatory-Initialize contract was the root cause of the entire factory-pre-init + NSD lazy-creation saga: LocalStorageNamedDeviceFactory.Get was forced to call device.Initialize(-1L) defensively just to satisfy the contract, which broke NativeStorageDevice (its single-shot Initialize then asserted on the consumer's follow-up Initialize(realSize)). Recent commits f4e3044 + 0535da0 papered over this with lazy native-handle creation; this commit removes the root cause. Changes: * StorageDeviceBase: remove the 'initialized' flag, EnsureInitialized() helper, and ThrowNotInitialized() method. Initialize() is now purely a *configuration* call to override the ctor defaults (set a non-default segment size, opt into OmitSegmentIdFromFileName). The ctor doc explicitly states that callers may issue IO immediately after construction. * All concrete devices: remove the EnsureInitialized() calls at the top of ReadAsync / WriteAsync / TruncateUntilSegmentAsync / RemoveSegment (libaio, io_uring, RA, ManagedLocal, LocalMemory, Null, Tiered, Sharded, Azure). * LocalStorageNamedDeviceFactory.Get: drop the defensive device.Initialize(-1L); the ctor defaults match what that call did anyway. * NSD's recent EnsureInitialized() additions to ReadAsync/WriteAsync (introduced in 0535da0 only to satisfy the hardening test) are also removed by the sweep. * ComponentRecoveryTests.cs: drop 3 redundant Initialize(-1L) calls. * test.hlog DeviceTests: rename and repurpose IDevice_*BeforeInitialize_Throws to IDevice_*BeforeInitialize_UsesCtorDefaults — the new test demonstrates that WriteAsync/ReadAsync on a freshly-constructed device (no Initialize call) works correctly using the unbounded single-segment defaults, across Native / RA / ManagedLocal. TestUtils.cs:165 still calls device.Initialize() — that path is conditional on the caller wanting OmitSegmentIdFromFileName=true, which IS only settable via Initialize (it is not a ctor parameter), so the call is genuinely needed there. Verified on Linux x64 / .NET 10: * libs/storage/Tsavorite/cs/test/test.hlog (IDevice + NativeStorageDevice tests): 62/62 pass. * libs/storage/Tsavorite/cs/test/test.recovery (ComponentRecovery tests): 4/4 pass. * Full Garnet.test, Garnet.test.cluster, Garnet.test.acl, Garnet.test.collections, Garnet.test.extensions, Garnet.test.scripting, Garnet.test.complexstring, Tsavorite IDevice+NSD: pass at the same rates as before this commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native + IDevice: refresh comments to reflect final design Sweep the PR for comments that referenced earlier design choices as they evolved during development, and rewrite them to describe the steady-state contract directly without historical baggage. * IDevice.Initialize: replace the "Must be called exactly once... before any IO entry point... uninitialized device throws InvalidOperationException" doc with the actual contract: Initialize is purely an opt-in configuration step to override the ctor defaults (which are equivalent to Initialize(-1)); callers may issue IO immediately after construction. * NativeStorageDevice ctor doc: rewrite to describe the as-shipped lazy creation flow (configuration captured at ctor, native handle created on first IO via EnsureNativeDeviceCreated) rather than the stale "Native device creation is DEFERRED until Initialize... every IO entry point throws InvalidOperationException" framing. * NativeStorageDevice.Initialize doc: drop the misleading "Creates the underlying native device with the requested segment size" lead-in (which hasn't been true since the lazy-creation refactor); replace the "factory pre-init" example (factory no longer pre-inits) with a steady-state description of when repeat Initialize calls are honoured. * NativeStorageDevice.EnsureNativeDeviceCreated doc: replace "Throws if Initialize has not been called" (now uses ctor defaults if no Initialize) with "Throws if the device has been disposed or if the native shim rejects the configuration". * NativeStorageDevice.EnsureReadyOrSilent doc: drop the "does not throw on 'not initialized yet'" qualification. * NativeStorageDevice.GetSectorSize doc: re-point the "cross-check" link from Initialize to EnsureNativeDeviceCreated (which is where it actually happens). * NativeStorageDevice.Dispose doc + body: bound the worst-case shutdown stall by the longest in-flight user callback (not CompletionWorkerTimeoutSecs) since wake-up uses NativeDevice_WakeCompletionWorker; rewrite the inline Dispose comment so it documents the steady-state design rather than what it improved over. * NativeStorageDevice nativeSegmentSizeBytes / UnboundedNativeSegmentSizeBytes field doc: clarify that the value is populated by EnsureNativeDeviceCreated (not Initialize) and that the default is reached without calling Initialize. * NativeStorageDevice_InitializeTwice_Idempotent test: drop the now-stale "factory pre-init... consumer re-initializes" rationale; describe the idempotent contract directly. * NativeStorageDevice_DisposeBeforeInitialize_IsNoOp test: drop the "Phase 6" reference and reword in terms of the steady-state lazy-creation contract. * SimulatedFlakyDevice.Initialize: replace the "so its EnsureInitialized() guard passes when our IO methods delegate to it" comment (the guard no longer exists) with a description of why both devices need matching geometry. * LinuxFileExtensions.OpenDirect dsync param doc: drop the "previously asked for it" wording — the WriteThrough callsites still pass it; describe the parameter as an opt-in for WriteThrough-equivalent semantics. * Doc-cref bookkeeping: change <see cref="base.segmentSize"/> (illegal cref for inherited fields) to <c>base.segmentSize</c> code spans. No behaviour change. Build clean on Garnet.slnx and Tsavorite.slnx; dotnet format --verify-no-changes clean on both. IDevice + NSD device tests all pass (62/62). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: ship libnative_device.so without liburing dependency Background: the shipped libnative_device.so was built with -DUSE_URING=ON, so it had a hard DT_NEEDED entry for liburing.so.2. Loading the .so on a host without liburing2 installed (e.g. a GitHub Actions ubuntu-latest runner, or an end-user box where only libaio is in the base image) failed with: System.DllNotFoundException: ... liburing.so.2: cannot open shared object file: No such file or directory …even for callers that only ever requested the libaio backend, because the dynamic linker resolves NEEDED libraries at load time regardless of which exported symbols the caller goes on to invoke. Rebuild the prebuilt with -DUSE_URING=OFF so the shipped .so links only libaio. Most Linux distributions ship libaio in the base system, so the prebuilt now loads without any additional setup. The io_uring backend becomes a build-time opt-in: callers that want it install liburing-dev and rebuild with -DUSE_URING=ON. The C# layer already surfaces a clear TsavoriteException for callers that request Uring against a USE_URING=OFF build ("Requested IO backend 'Uring' is not available in the loaded native_device library… Rebuild the native library with -DUSE_URING=ON and install liburing-dev to enable io_uring."). Build fix: file_linux.h now includes <fcntl.h> directly (for the ::open() / O_RDONLY usage in QueueIoHandler::OpenWakeFd()). Previously these were pulled in transitively through <liburing.h>, which is now gated behind #ifdef FASTER_URING. Dockerfile updates: drop liburing2 / liburing from the runtime install list in all 5 Dockerfiles (default, .ubuntu, .alpine, .azurelinux, .chiseled). Comments left for users that rebuild with USE_URING=ON. README updates: rewrite the "Runtime dependencies" section to describe the new default (libaio only). Replace the "Disabling io_uring (optional)" section with "Enabling io_uring (optional)". Verified on Linux x64 / .NET 10: libaio default works (62/62 IDevice + NativeStorageDevice tests pass); ldd confirms only libaio.so.1t64 is in NEEDED. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: ship two .so flavors — uring-enabled + libaio-only fallback Single-shipping libnative_device.so created a deployment dilemma: build with USE_URING=ON and end-users without liburing get DllNotFoundException at load time; build with USE_URING=OFF and the io_uring backend stops working even on hosts that DO have liburing installed (which is the case where uring matters for perf — modern NVMe at >1M IOPS benefits noticeably from uring over libaio). Ship both flavors instead: * libnative_device.so — built with USE_URING=ON; DT_NEEDED on libaio AND liburing. Exposes both Libaio and Uring backends. * libnative_device_libaio.so — built with USE_URING=OFF; DT_NEEDED on libaio only. Exposes the Libaio backend. NativeStorageDevice's DllImportResolver tries the uring-enabled binary first; on DllNotFoundException matching 'liburing.so.2: cannot open' it falls back to the libaio-only binary. The Libaio backend therefore always works out of the box on any Linux distribution that ships libaio (essentially all of them). liburing is opt-in: hosts that install it get the Uring backend with zero runtime overhead vs Libaio (direct calls, no function-pointer indirection — we deliberately rejected the dlopen approach so the future-default uring path stays optimal). If a caller explicitly selects IoBackend.Uring on a host without liburing, the construction-time error message now points at the install command per distro ('apt-get install -y liburing2', 'dnf install -y liburing', 'apk add liburing') instead of telling the user to rebuild the .so with -DUSE_URING=ON. We never silently downgrade Uring to Libaio. Changes: * NativeStorageDevice.cs: new LibaioFallbackLibraryPath; ImportResolver catches DllNotFoundException for liburing.so.2 and falls back to the libaio-only .so. ResolveNativeLibraryPath now takes the path as a parameter so it can resolve either flavor. * NativeStorageDevice.cs: rewrite the 'backend not available' exception message — point at install commands (the actual remediation) not rebuild. * Tsavorite.core.csproj: add libnative_device_libaio.so as a second ContentWithTargetPath asset so both .so files are copied to the output directory and packed into the NuGet runtime payload. * runtimes/linux-x64/native/libnative_device.so — REPLACED with USE_URING=ON build (DT_NEEDED libaio + liburing). 2.3 MB. * runtimes/linux-x64/native/libnative_device_libaio.so — NEW, USE_URING=OFF build (DT_NEEDED libaio only). 1.6 MB. * Dockerfile, Dockerfile.ubuntu, Dockerfile.alpine, Dockerfile.azurelinux, Dockerfile.chiseled: re-add liburing2 / liburing to the runtime installs so docker users get the io_uring backend out of the box (the libaio-only fallback would otherwise leave Uring unusable inside containers). * cc/README.md: rewrite the 'Runtime dependencies' and build sections to describe the two-flavor layout, drop the stale 'Enabling io_uring' section, and document the prebuilt rebuild workflow. Verified end-to-end: * Both backends saturate the Dell P5600 NVMe at ~743K random read IOPS in benchmark/Device.benchmark (matches the pre-change reference). * 62/62 IDevice + NativeStorageDevice tests pass. * dotnet format clean on both Garnet.slnx and Tsavorite.slnx. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native (Windows): add init_errno()/initialized() stubs to ThreadPoolIoHandler NativeDeviceImpl's constructor (native_device.h:100-101) gates on handler_.init_errno() to surface an actionable error message when the underlying IO handler failed to initialize (e.g., libaio io_setup() failed with EMFILE / ENOMEM, or io_uring_queue_init() failed). The Linux handlers QueueIoHandler and UringIoHandler both expose this API; ThreadPoolIoHandler (Windows) did not, so MSVC failed to instantiate NativeDeviceImpl<ThreadPoolIoHandler> with: error C2039: 'init_errno': is not a member of 'FASTER::environment::ThreadPoolIoHandler' Add init_errno() and initialized() stubs that return 0 / true unconditionally — the Windows ThreadPool API does not have a separable init step that can fail in the same way the Linux io_setup / io_uring_queue_init paths can (threadpool creation failures propagate via threadpool_'s ctor, not via a later 'check this' field on the handler), so the stubs are semantically correct. NativeDeviceImpl then falls through to the log_.Open(&handler_) path which is where Windows-specific errors (missing directory, permission denied, etc.) actually surface. Linux unaffected: rebuilt build/Release-uring cleanly after this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native tests: stop skipping NSD tests on Windows The Native tests in DeviceTests.cs had blanket 'NativeStorageDevice is Linux-only' Assert.Ignore guards that dated from when NSD's C++ shim was Linux-only. The shim is built on Windows too (native_device.dll via the ThreadPool / IOCP backend in file_windows.cc), so directly constructing 'new NativeStorageDevice(...)' works on Windows. The blanket guards were silently dropping ~15 NSD test cases on Windows CI. Drop the guards so the tests exercise the Windows C++ shim. The legitimate Linux-only guard on IDevice_PermissionDeniedAtFirstWrite_CallbackGetsError (chmod-based; chmod has no Windows analogue) is preserved. Important: end-user device routing is UNCHANGED. Devices.CreateLogDevice(DeviceType.Native) on Windows still returns LocalStorageDevice (managed Windows IOCP), not NativeStorageDevice — that routing happens in Devices.cs and was not touched. These tests directly instantiate the NSD class for shim-coverage purposes only; they do not affect what end users get from the default device factory. Linux: 62/62 pass after this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: rebuild win-x64 native_device.dll for the latest C++ source Rebuild the shipped Windows prebuilt from the current native_device source so the latest fixes (Initialize idempotence, WakeCompletionWorker, etc.) are reflected in the win-x64 DLL. USE_URING is a no-op on Windows; the DLL only exposes the Default (IOCP) backend, so there is no equivalent of the libnative_device_libaio.so fallback on this platform. End-user device routing is unchanged: Devices.CreateLogDevice(DeviceType.Native) on Windows still returns LocalStorageDevice (managed Windows IOCP). This DLL is exercised by direct 'new NativeStorageDevice(...)' construction (see Tsavorite.test.hlog DeviceTests — 59/59 Native + IDevice tests pass on Windows after this rebuild). Built with: Visual Studio 17 2022, MSVC v143, x64, Release configuration, Spectre-mitigated CRT. * Tsavorite Native: address GPT-5.5 PR review findings Fixes two correctness issues caught by an automated code review of the optimize-device PR. ### io_uring SQE leak on submit failure In UringFile::ScheduleOperation, io_uring_get_sqe() advances the user-side sqe_tail before io_uring_submit() is called. If submit fails after retries (-EAGAIN/-EBUSY exhausted, or any other negative), the old code released the lock and returned IOError without doing anything about the still-pending SQE. user_data on that SQE pointed at the io_context unique_ptr that was about to be freed by the guards unwinding, so the next successful submit on the same ring would consume the stale SQE and the QueueRunFor drain loop would dispatch a callback against freed memory — a clear use-after-free. Fix: before releasing sq_lock on the failure path, rewrite the still- pending SQE in place as io_uring_prep_nop with user_data = nullptr. The drain loop already skips nullptr user_data (it's the wake-up sentinel used by UringIoHandler::Wake), so when a later submit flushes this nop the CQE is drained harmlessly. Safe to mutate the SQE in place because we still hold sq_lock and no kernel/concurrent submitter has observed it yet. ### NativeDevice sector_size always returned 512 FileSystemSegmentedFile::alignment() returned a hard-coded 512. NativeDeviceImpl::sector_size() delegated to it, so the C# wrapper's sector-size cross-check in EnsureNativeDeviceCreated would: - falsely throw on 4K-native disks where ProbeAlignment returns 4096 (managed 4096 vs native 512 → 'sector-size mismatch' → device unusable), or - on 4K-native disks where the managed probe fell back to 512 (e.g. older kernel without STATX_DIOALIGN), let the device initialize with SectorSize=512 and then have the kernel reject the 512-aligned O_DIRECT buffers with EINVAL. Fix: factor the STATX_DIOALIGN probe from NativeDevice_ProbeAlignment into a shared inline helper (native_device::ProbeDioAlignment in native_device.h) and call it once from the NativeDeviceImpl ctor, caching the result as the immutable member device_alignment_. sector_size() now returns the cached value; NativeDevice_ProbeAlignment delegates to the same helper. Both sides of the ABI cross-check go through identical probe logic, so the check is now a meaningful ABI / runtime-drift detector instead of a 4K-disk footgun. ### Stale IDevice.Initialize XML The omitSegmentIdFromFilename param said it was 'only supported by managed devices — NativeStorageDevice rejects this flag'. Native devices have honored the flag since 6584cf7. Updated the doc. ### Alpine install hint The 'IoBackend.Uring with libaio fallback' error message suggested 'sudo apk add liburing' on Alpine, but README.md notes that the prebuilt won't load on Alpine (musl) at all. Replaced the apk suggestion with the actual Alpine support story (use a glibc image or fall back to a managed device). Verification: 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass on Linux. Both .so binaries rebuilt (uring-enabled and libaio-only fallback) with correct ldd output. Device.benchmark NVMe saturation throughput unchanged within noise (libaio 738K IOPS, uring 349K IOPS on Dell P5600). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: probe sector size via sysfs max(logical, physical) Replaces the statx(STATX_DIOALIGN) probe in ProbeDioAlignment with a direct sysfs lookup of max(logical_block_size, physical_block_size). Why: - STATX_DIOALIGN reports only the kernel-enforced minimum (= logical block size). It misses the firmware's preferred sector (physical_block_size), so on a 512e drive (logical=512, physical=4096) the probe would return 512 and Tsavorite would take a firmware RMW penalty on every partial-sector write. - STATX_DIOALIGN also requires kernel 6.1+ AND the filesystem to populate the field; ext4 on 6.8 leaves it unset on 512-byte devices, so the probe was already falling through to the 512 default in practice. - sysfs gives us both values directly, on every kernel, with no O_DIRECT dance. Taking max(logical, physical) covers the correctness floor (logical = kernel-enforced minimum) and the performance floor (physical = avoid RMW on partial writes) in one shot. Implementation: - stat() the file (or its closest existing ancestor — log file may not exist yet at construction). Extract st_dev → (major, minor). - Read /sys/dev/block/<maj>:<min>/queue/{logical,physical}_block_size. For partitions (e.g. sda2), the queue/ dir lives on the parent whole-disk block device — fall through to ../queue/<field>. - Round result up to a power of two (always already pow2 on real hardware) and floor at 512 B. On this machine (Dell P5600 NVMe + PERC sda): Probe(/DATA2/badrishc) = 512 (NVMe: logical=512, physical=512) Probe(/tmp/devbench) = 512 (sda partition via parent-walk) Probe(/home/badrishc) = 512 Probe(/) = 512 All values match max(logical, physical) read directly from sysfs. Verification: - 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass - Both .so flavors rebuilt (uring-enabled + libaio-only fallback) - C ABI NativeDevice_ProbeAlignment delegates to the same helper, so managed SectorSize and native sector_size() remain in lockstep. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native tests: align test buffers to 4096, matching sysfs-probed sector size CI failure on IDevice_Initialize_OmitSegmentIdFromFilename_BareFileName (and likely other IDevice_* tests on the same runner) reported: 'NativeStorageDevice.WriteAsync: misaligned I/O — sector size is 4096, but offset=0x0, length=4096, buffer=0x...7EC7A9F76800' The buffer ends at 0x...800 = 2048 — i.e. 2048-aligned but not 4096-aligned. The test helper allocated buffers aligned to HardeningSectorSize = 512 (the pre-PR default for every Garnet Linux device); a 512-aligned formula can land on a 2048-boundary that is not also a 4096-boundary. CI's underlying disk reports physical_block_size = 4096 in sysfs, so the new max(logical, physical) probe returns 4096 there. The native shim then correctly rejects sub-4096-aligned O_DIRECT buffers with EINVAL. The fix is on the test side: bump HardeningSectorSize from 512 to 4096 so the test buffer alignment matches the strictest device.SectorSize seen on any modern hardware (512n, 512e, 4Kn). Locally (Dell P5600 NVMe, logical=physical=512 → SectorSize=512) all 62 IDevice + NativeStorageDevice tests still pass — 4096 trivially divides 512. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite: revert GetAndPopulateReadBuffer changes (defer to separate PR) The read-window sizing optimization (drop leading-slop padding + clamp to page-end) is out of scope for this device-backend PR; it interacts with the larger read-IO path and deserves its own focused PR with dedicated benchmarking. Reverting to the pre-PR behavior here. TryAllocateRetryNow's bounded-backoff change is retained — it's a self-contained allocator hot-path fix and is independently verified (+13.9% on YCSB load with libaio at 64 threads, per kvbench benchmarking). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: Windows probe uses IOCTL_STORAGE_QUERY_PROPERTY for max(logical, physical) Symmetry with the Linux sysfs probe — Windows now reads both BytesPerLogicalSector and BytesPerPhysicalSector from the volume's STORAGE_ACCESS_ALIGNMENT_DESCRIPTOR (via IOCTL_STORAGE_QUERY_PROPERTY + StorageAccessAlignmentProperty) and returns the rounded-up-to-pow2 max, floor 512 B. Previously the Windows branch returned 512 unconditionally, which would silently undersize SectorSize on Windows 4Kn / 512e drives. Implementation: - Parse drive letter from filename ("C:\foo.dat" -> "\\.\\C:"). UNC paths are not supported by this probe — fall back to 512. - CreateFile on the volume with FILE_READ_ATTRIBUTES (no admin needed). - DeviceIoControl(IOCTL_STORAGE_QUERY_PROPERTY) with StorageAccessAlignmentProperty. - max(logical, physical), round up to pow2, floor 512. Linux behavior unchanged. Both .so flavors rebuilt and pass 62/62 device tests on this machine (logical=physical=512 NVMe). REQUIRES Windows DLL rebuild — the Windows path in ProbeDioAlignment is now non-trivial, and the existing prebuilt native_device.dll still returns 512 unconditionally. Without the rebuild, on a Windows 4Kn box the managed SectorSize cross-check would (incorrectly) pass at 512 while the device might actually need 4096. Rebuild recipe in the companion review comment / Tsavorite/cc/README.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add binaries * Tsavorite Native: skip wake-up/failed-submit sentinel CQEs in TryCompleteFor Addresses Copilot review comment on file_linux.cc:373. UringIoHandler::TryCompleteFor (and via it TryComplete) dispatched every drained CQE through DispatchUringCqe without checking the user_data = nullptr sentinel that QueueRunFor already handles. The sentinel marks two kinds of no-op CQEs: - Wake-up nops submitted by UringIoHandler::Wake to unblock the drainer on Dispose. - SQEs rewritten in-place after io_uring_submit failed (the SQE leak fix in c6d68925); these are committed to the SQ but carry no caller context. If a TryComplete() / TryCompleteFor() call picks up either kind of nop CQE, DispatchUringCqe would dereference the null context at context->callback(...) and segfault. Fix: mirror the nullptr-skip from QueueRunFor in TryCompleteFor. Return true to count the drain (matching the any-flag semantics). Verification: 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass. uring .so rebuilt; libaio-only .so is byte-identical because the patched code path is wrapped in #ifdef FASTER_URING and not compiled into the libaio-only fallback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native (uring): batch-drain CQEs and dispatch outside cq_lock QueueRunFor used to acquire cq_lock per CQE (peek -> read fields -> cqe_seen -> release -> dispatch). With a single drainer thread that serializes lock acquire/release on every completion and forces submitters to wait through callback latency when they need ring access. Replaced with the canonical liburing batch-drain idiom: - acquire cq_lock once - io_uring_peek_batch_cqe(ring, cqes, 64) to pull up to 64 CQEs - snapshot (io_res, context) for each - io_uring_cq_advance(ring, n) to release the slots - release cq_lock - dispatch callbacks outside the lock This is the io_uring equivalent of libaio's io_getevents(n) per syscall. Snapshot BEFORE cq_advance is mandatory because the kernel may reuse CQ slots once advanced, leaving the cqe pointers dangling. The wake-up / failed-submit sentinel (user_data == nullptr) is still skipped without dispatch, same as before. Measured impact on Dell P5600 (16 submitter threads, batch 64, throttle 256): ct=1 (1 ring, 1 drainer): 339K -> 354K ops/sec (+4%) ct=4 (4 rings, 4 drainers): 735K -> 737K (saturates, noise) ct=8 (8 rings, 8 drainers): 750K -> 742K avg (saturates, noise) The single-drainer gain is modest because the real bottleneck at ct=1 with 16 submitters is sq_lock contention on the single ring, not cq_lock contention. The batch-drain is still strictly better: - dispatches outside the lock so submitters aren't blocked by user-callback latency, - matches the idiomatic liburing pattern, - amortizes the lock acquire/release across up to 64 CQEs per cycle. For high-throughput workloads, sharding across multiple rings remains the right scaling lever (ct >= 4 saturates this drive). Verification: 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass. uring .so rebuilt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native (uring): per-thread ring affinity + 4 default rings Eliminates the sq_lock contention that was capping uring at ~340K IOPS at the default numCompletionThreads=1. Two changes work together: 1. Per-thread ring affinity in pick_ring (file_linux.h): Each submitter thread is assigned a ring on its first submit (round- robin against other threads via an atomic counter) and keeps that assignment for life. Same-thread submits never contend on sq_lock with themselves; different threads only contend when they got assigned the same ring (num_submitter_threads > num_rings). This is the user-space equivalent of libaio's "io_submit is thread-safe per io_context" — eliminate shared mutable state across submitters. 2. Hardcoded 4 rings for uring (NativeStorageDevice.cs): numIoContextsConfig = ioBackend == Uring ? max(kDefaultUringRings=4, numCompletionThreads) : numCompletionThreads So uring always has at least 4 rings even at numCompletionThreads=1. The single drainer covers all 4 rings via the legacy QueueRun compat scanner (CompletionWorker passes ctxIdx=-1 in that case). libaio is unchanged: rings == numCompletionThreads (extra rings don't help; the kernel io_context mutex is already efficient). Result on Dell P5600 NVMe (16 submitter threads, batch 64, throttle 256): Before (1 ring, 1 drainer): ~340K After (4 rings, 1 drainer, default): ~700K (matches libaio ct=1) After (8 rings, 8 drainers, sharded): ~745K (unchanged, was already saturating) No new public configuration parameters. numCompletionThreads still controls drainer count; the ring count is now backend-derived behind the scenes. The CompletionWorker single-drainer-multi-ring path was added specifically so the default numCompletionThreads=1 case can saturate without spawning extra drainer threads. Also: bumped HardeningSectorSize and the legacy bufferPool / NativeDeviceTest2 sector_size constants from 512 to 4096 to match the strictest device SectorSize we expect on any modern hardware (4Kn drives where the new max(logical, physical) probe returns 4096). Tests would otherwise fail with EINVAL on 4Kn CI runners with 512-aligned buffers. Verification: - 64/64 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass - Default uring (no flags) hits 626-740K across t=1..64 vs ~340K before - Sharded ct=4/8 unchanged (still saturates) - libaio default unchanged Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add binaries * Tsavorite Native tests: fix ReadInto length mismatch surfaced by 4096 SectorSize NativeDeviceTest1 read 1024 bytes (entryLength) using ReadInto, which: - rounded the read length up to the device sector size, - then returned a buffer of that ROUNDED length, - which the caller compared via SequenceEqual against the original `entry` byte[] (length 1024). When SectorSize was 512 (the old constant probe), 1024 rounded to 1024 and the lengths happened to match. With the new max(logical, physical) probe returning 4096 on 4Kn drives (Windows/Ubuntu CI runners), 1024 rounds to 4096, the returned buffer is 4096 bytes long, and SequenceEqual fails on length mismatch (regardless of content). Pre-existing latent bug — the rounding to sector size is correct for the IO submit, but the caller should only see the bytes it asked for. Fix: return a buffer of the caller-requested logical `size`, not the sector-rounded `numBytesToRead`. Verification: 64/64 Tsavorite.test.hlog NativeDeviceTest + IDevice + NativeStorageDevice tests pass on Linux (where SectorSize is 4096 on the CI runner's 4Kn drive). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite benchmarks: refresh stale completion-threads help text The --device-completion-threads (KV.benchmark) and --completion-threads (Device.benchmark) help text said "all drainers share the same kernel io_context / io_uring" and "values > 1 are rarely useful past 1 today". Both claims are stale since the sharded-rings work (8cbca9d4d) and the per-thread ring affinity + 4-default-rings change (298bfd180): - Each drainer is bound 1:1 to its own kernel io_context (libaio) or io_uring ring (uring). - Submitters distribute across rings via per-thread affinity. - For io_uring, throughput scales with completion-threads up to available submitter concurrency (measured: ct=1 ~340K → ct=4 ~735K on Dell P5600 NVMe at the device-benchmark level). - For libaio extra drainers still rarely help past 1 (kernel per-context mutex is efficient). - Note added that uring uses min 4 rings even at ct=1 with the single drainer covering all rings via the legacy QueueRun scanner. Help-text-only change. No code behavior change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: fix KV.benchmark deadlock on multi-segment disk-spill reads Root cause: cross-segment read rejection + engine retry loop ============================================================== The AllocatorBase.GetAndPopulateReadBuffer sector-aligned read window can extend past the page-end boundary when reading a record near the tail of a page. When the device's segment size is a multiple of the page size (e.g. 4MB pages, 1GB segments — the Garnet default), an over-extended read at the last page of a segment also crosses the device's segment boundary. NativeStorageDevice's underlying FileSystemSegmentedFile rejects cross-segment reads with Status::IOError; the engine's AsyncGetFromDiskCallback interprets a 0-byte read as a short read and retries the same address — forever. Worker thread spins at 99% CPU, disk activity drops to zero, benchmark deadlocks. Reproduced reliably on KV.benchmark: --device native --device-io-backend libaio \ --log-memory 16m --page-size 4m --segment-size 1g \ -n 10000000 (1.28 GB dataset → crosses 1GB segment boundary) Smaller datasets (1M = 128MB, fits in 1 segment) work; larger ones hang. RandomAccess device works on all dataset sizes because its managed segmented-file wrapper doesn't reject cross-segment reads. Diagnostic captured the exact symptom: a read at sourceAddress 0x3FFFF600 (1,073,739,776 — 2,560 bytes before the 1GB segment boundary) with readLength 4608 (sector-aligned record window) extends to 0x40000C00 — 2,560 bytes into segment 1. Native rejects with Status::IOError, callback fires with numBytes=0, engine retries. Fix: clamp the aligned read length so it never crosses page-end. ============================================================ Added in AllocatorBase.GetAndPopulateReadBuffer: var pageEndInFile = (ulong)(AlignedPageSizeBytes * (GetPage(fromLogicalAddress) + 1)); if (alignedFileOffset + alignedReadLength > pageEndInFile) alignedReadLength = (uint)(pageEndInFile - alignedFileOffset); Records never span page boundaries (HandlePageOverflow guarantees), so the actual record is fully readable within the clamped window — available_bytes reflects what we actually got from disk, and the engine continues normally. pageEnd is sector-aligned (PageSizeBits >= sector size), so the clamped length stays sector-aligned. Also reverted the uring "min 4 rings even at ct=1" experiment ============================================================= The earlier "default 4 rings for uring regardless of ct" change was fundamentally broken: with per-thread submit affinity (pick_ring's thread_local index), submitters bound to rings 1-3 never get their completions drained because the single drainer blocks on ring 0 with a 1-second QueueRun timeout and only briefly polls the other rings between wake-ups. The result is ~50x throughput degradation on workloads where submitters land on rings != 0 (KV.benchmark load phase dropped from 2.5M ops/sec to 54K ops/sec at t=1). Reverted to the simple rule: rings == numCompletionThreads. For uring perf scaling, users set numCompletionThreads >= expected submitter concurrency; each ring is then continuously drained by its dedicated drainer thread. Defense-in-depth hardening ========================== - NativeStorageDevice._callback now catches ALL exceptions from the user callback (was: try/finally but exception propagated). A managed exception escaping back into native code across the C ABI boundary silently terminates the drainer thread; the next submitter then spins forever in device.Throttle(). Now the exception is logged and swallowed so the drainer survives. - NativeStorageDevice.CompletionWorker has the same try/catch around the whole drain loop as defense-in-depth against unrelated managed exceptions (P/Invoke marshalling, IntPtr.Zero races with Dispose, etc.). - file_linux.cc QueueFile::ScheduleOperation (libaio) and UringFile::ScheduleOperation (uring) now retry submit-side EAGAIN indefinitely with bounded backoff (64 sched_yields, then 1ms nanosleeps) instead of returning Status::IOError after 8 yields. Surfacing transient EAGAIN as a permanent error creates the same retry-loop pathology as the cross-segment-read bug above. EAGAIN is the kernel saying "ring is full, try later"; it's not a real error and must not be exposed to the engine. Verification ============ KV.benchmark, 100M keys × 100B, 16MB log (mostly disk-spill), 1 completion thread, 100% reads: libaio: t=1 135K ops/sec, t=4 400K, t=8 444K, t=16 445K, t=32 404K uring: t=1 124K ops/sec, t=4 244K, t=8 265K, t=16 278K, t=32 272K Both backends stable across the full thread × dataset sweep (previously native+libaio hung on any 10M+ dataset; native+uring hung on every config). 64/64 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> | 3 个月前 | |
[Tsavorite] Add native Linux storage backend, harden NativeStorageDevice, refresh storage benchmarks (#1831) * Port device/IO changes from optimize-v2-io onto kv-bench All 31 non-benchmark files changed on optimize-v2-io (vs its branch base d3677cfaa) ported here. Backup tag: optimize-v2-io-prerebase-backup @ 3f41f2bdf. Scope: device/IO/native-backend ONLY. Includes: Tsavorite C++ native device: - io_uring backend + pluggable C ABI (file_linux.cc/h) - error model split (native_device_error.h) - file_system_disk + native_device.h updates - CMakeLists + README Tsavorite C# device: - NativeStorageDevice: IoBackend enum (Default, Libaio, Uring), completion threads, production-readiness pass - LinuxFileExtensions.cs: P/Invoke open() for true O_DIRECT - ManagedLocalStorageDevice + RandomAccessLocalStorageDevice: O_DIRECT wiring on Linux - Devices.cs: router updates for new device APIs Tsavorite allocator + utilities: - AllocatorBase: bounded backoff in TryAllocateRetryNow - CompletionEvent: Wait(TimeSpan) overload Tsavorite checkpoint management: - LocalStorageNamedDeviceFactory + Creator surface ioBackend + completionThreads parameters Tests: - DeviceTests.cs updated for new device APIs Garnet host: - --device-io-backend, --device-completion-threads flags - defaults.conf updated, GarnetServerOptions wiring Dockerfiles (all 5): install liburing alongside libaio. Note: YCSB.benchmark and KV.benchmark are not modified by this commit; KV.benchmark is the supported benchmark on this branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * KV.benchmark: add uring backend, fix --validate for disk-spill, doc liburing runtime dep - Options: --device-io-backend now accepts 'uring' (aliases: io_uring, iouring) in addition to libaio/default; help text + validation error list updated to match. - KV.benchmark README: device-backend table now describes the libaio vs uring split, with a runtime-install snippet for liburing across Debian/Ubuntu, Fedora/RHEL/AzureLinux, and Alpine, plus link to the full Tsavorite Native Device docs. New 'native + libaio' and 'native + uring' rows added to the cookbook for constrained-log large-dataset workloads. - Tsavorite Native Device README: new top-level 'Runtime dependencies (end users)' section listing the apt/dnf/apk install lines and how to fall back to the no-liburing variant. - KV.benchmark Validate: fix two bugs that surface when load and run use different thread counts and when the log spills to disk: 1) writerThread reconstruction now uses ResolvedLoadThreads (not Options.Threads which is the RUN count), so --load-threads N with --threads M != N validates correctly. 2) Reads of records below HeadAddress return Status.IsPending; the previous code counted these as misses. Validate now issues reads in batches of 256 and drains via CompletePendingWithOutputs, verifying each completed output against the per-thread pattern. Verified end-to-end with both backends: 4.6M × 100B × 8T × log=256m (~580MB dataset > 256MB log → forces disk spill) × 50R/50U × --validate: native + libaio → [validate] OK, run = 1.27 M ops/s native + uring → [validate] OK, run = 1.02 M ops/s 4.6M × 100B × 8T × log auto (fits) × 95R/5U × --validate: native + libaio → [validate] OK, run = 16.10 M ops/s native + uring → [validate] OK, run = 16.33 M ops/s Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite: replace MarkHandleAsAsync reflection hack with RandomAccess on SafeFileHandle Background: MarkHandleAsAsync used reflection to flip SafeFileHandle.IsAsync's non-public setter so a P/Invoke-opened O_DIRECT FD could be wrapped in 'new FileStream(handle, isAsync: true)' without throwing 'Handle does not support asynchronous operations'. The flag is non-public in .NET 8/10, the hack was fragile across future runtime versions, and on Linux IsAsync is a contract gate (no real overlapped I/O exists for files), so the lie bought us nothing beyond letting the FileStream constructor accept the handle. RandomAccessLocalStorageDevice — refactor: - StorageAccessContext.handle is now SafeFileHandle (was FileStream). - CreateRead/WriteHandle: * Linux + O_DIRECT capable: LinuxFileExtensions.OpenDirect -> raw SafeFileHandle, no FileStream wrap. Page-cache bypass via the O_DIRECT flag at open(2), exactly as before. * Otherwise (Windows; or Linux when filesystem rejects O_DIRECT): File.OpenHandle(path, ..., FileOptions.Asynchronous | cast FILE_FLAG_NO_BUFFERING). On Windows this gives the runtime IOCP-bound OVERLAPPED I/O; on Linux it's page-cached. - All I/O goes through RandomAccess.{Read,Write}Async(safeHandle, memory, offset). On Windows: true kernel async via IOCP. On Linux: pread/pwrite dispatched to ThreadPool (same as before). - GetFileSize uses RandomAccess.GetLength(handle). - SetFileSize uses RandomAccess.SetLength(handle, size). LinuxFileExtensions: - MarkHandleAsAsync and the IsAsyncProperty reflection are deleted entirely (no remaining callers). - System.Reflection using removed. ManagedLocalStorageDevice: - Reverted to origin/main. This device is designed to stay within FileStream APIs; the O_DIRECT branch we added doesn't belong here. Verified: - Tsavorite test.hlog DeviceTests: 36/36 passed. - KV.benchmark --device randomaccess --log-memory 256m --preallocate-log --rumd 50,50,0,0 --validate: PASS, 357 K ops/sec, iostat shows 100-177 K real disk r/s and 53-90% NVMe util → O_DIRECT page-cache bypass confirmed. - KV.benchmark --device randomaccess (log fits) --validate: PASS, 15.7 M ops/sec. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite tests: cross-device hardening suite (Native + RandomAccess + ManagedLocal) The Phase-7 hardening suite added in this branch was NativeStorageDevice- specific. Add parametrized variants of the four tests that are pure IDevice contract checks (not native-specific lifecycle/API), so they exercise all three local-storage device implementations: - Hardening_AllDevices_RoundTrip_BasicReadWrite - Hardening_AllDevices_RoundTrip_AcrossSegmentBoundary - Hardening_AllDevices_Parallel_32ConcurrentWrites - Hardening_AllDevices_Parallel_BurstyTraffic Each is parametrized by a new DeviceKind enum (Native, RandomAccess, ManagedLocal). Native is gated on OperatingSystem.IsLinux() (the C++ shim links against libaio/liburing); the other two run on both Linux and Windows. A shared CreateDeviceForTest helper takes care of the per-kind ctor + Initialize() dance so the test body stays uniform. Result: 38/38 hardening tests pass on Linux (12 new cross-device + 26 native-only). Native-specific tests retained as-is because they test API that doesn't exist on the other devices: - Lifecycle (DisposeBeforeInitialize, InitializeTwice, etc.) — NativeStorageDevice defers Initialize from the ctor; the other devices initialize in their ctor. - Segment-size validation (NonPowerOfTwoSegmentSize_Throws, etc.) — Initialize() is the only callsite that validates. - Recovery_*_SegmentSize_* — the native device's open() path is the only place that records and re-validates per-segment-size metadata. - SectorSize stability across opens — not all devices expose this. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite device tests: split into IDevice_ contract + NativeStorageDevice_ buckets; fix AsyncPool creator-throw hang Test refactor: - IDevice_*: 8 contract tests parametrized across Native, RandomAccess, ManagedLocal (round-trip basic, round-trip cross-segment, round-trip various segment sizes, 32 concurrent writes, 64 concurrent reads, mixed reads+writes, bursty traffic, stress burst of 100 writes, permission-denied callback contract). 33 cases total. - NativeStorageDevice_*: 16 native-only tests for behaviors managed devices don't have (deferred Initialize signature, recovery segment-size mismatch detection, sector-size discovery, sync-throw unaligned IO guard). AsyncPool fix: GetOrAdd reserved a slot in totalAllocated before calling creator(). If creator() threw (e.g. open() returned EACCES, ENOSPC), the slot was never released, so Dispose() would loop forever waiting for totalAllocated to drain to zero. This manifested as a process hang when a device pool's first open() failed. Rollback the reservation on exception so the failure propagates cleanly and the pool can still be disposed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite IDevice: unify Initialize contract — required for all devices, -1 = unbounded single segment Before this change, IDevice.Initialize had the same signature for every device but very different semantics: * StorageDeviceBase (RandomAccess, ManagedLocal, LocalStorage, NullDevice, LocalMemoryDevice): the ctor pre-set segmentSize = -1 / bits = 64 / mask = ~0, so calling an IO entry point without Initialize() silently ran in unbounded single-segment mode. * NativeStorageDevice: Initialize was MANDATORY (the C++ shim needs the segment size at create time for libaio/io_uring geometry), IO entry points threw if invoked first, and segmentSize = -1 was rejected. This commit unifies the contract: every IDevice must call Initialize() exactly once before any IO, and segmentSize = -1 selects unbounded single-segment mode on every device. Implementation: * StorageDeviceBase - Added `initialized` flag (volatile) and `EnsureInitialized()` helper that throws InvalidOperationException with a clear message naming the device by FileName. - Ctor leaves `initialized = false` but keeps the safe fallback defaults (-1 / 64 / ~0) so any cold maintenance path that touches segmentSizeBits before the guard can't compute outright nonsense. - EnsureInitialized() called from the base address-based ReadAsync / WriteAsync overloads and TruncateUntilAddress / TruncateUntilAddressAsync. - Initialize sets `initialized = true` at the end. * NativeStorageDevice - Accepts segmentSize = -1: translates to 1UL << 63 for the native shim so the C++ FileSystemSegmentedFile's shift = log2(segment_size) math collapses every non-negative upper-layer address into segment 0 (parity with the managed-side bits = 64 / mask = ~0). Single growing file on disk. - Tracks the value passed to native in nativeSegmentSizeBytes (replaces the diagnostic-only configuredSegmentSizeBytes long field, which couldn't hold 1<<63 without overflow). - ABI readback (NativeDevice_GetSegmentSize) compared against the value we sent to native, not the user-facing -1. - Always rejects omitSegmentIdFromFilename — the C++ shim has no omit-suffix code path, every segment is written as <base>.<segmentId>. Better to fail fast than silently produce wrong file names. * Concrete IO entry points (ReadAsync / WriteAsync / RemoveSegment / RemoveSegmentAsync) of NullDevice, LocalMemoryDevice, ManagedLocalStorageDevice, RandomAccessLocalStorageDevice, LocalStorageDevice, AzureStorageDevice, ShardedStorageDevice, and TieredStorageDevice now call EnsureInitialized() before doing work. Caller fix-ups: * LocalStorageNamedDeviceFactory.Get now calls device.Initialize(-1L) before returning. Commit / checkpoint metadata is single growing-file usage (segment 0 only, .0 suffix), so unbounded mode is the right default and unblocks every DeviceLogCommitCheckpointManager caller from needing to remember to initialize. * LocalStorageNamedDeviceFactory.ListContents skips dotfile entries — defensive against a pre-existing race in LinuxFileExtensions.IsDirectIOSupported where a .tsavorite-odirect-probe-* temp file can leak in the commit dir if File.Delete races with File.GetFiles. Without this filter, leaked probe files surface as Int64.Parse("") failures in DefaultCheckpointNamingScheme.CommitNumber. * SimulatedFlakyDevice.Initialize now propagates to the wrapped device. * ComponentRecoveryTests Setup_* helpers call Initialize(-1) on devices they construct directly (bypass the Tsavorite allocator path which normally Initializes). Tests (DeviceTests.cs): * IDevice_ReadAsyncBeforeInitialize_Throws(kind) × 3 — new contract test. * IDevice_WriteAsyncBeforeInitialize_Throws(kind) × 3 — same. * IDevice_Initialize_SegmentSizeMinusOne_UnboundedSingleSegment(kind) × 3 — write at offset 1 MiB (would be in segment-N for any positive size) and read back, confirming -1 routes through segment 0 on all 3 kinds. * NativeStorageDevice_Initialize_OmitSegmentIdFromFilename_Throws — new native-only test for the omit rejection in both -1 and explicit-size modes. * Removed NativeStorageDevice_{Read,Write}AsyncBeforeInitialize_Throws (now subsumed by the IDevice_ variants). Docs: * IDevice.Initialize docstring rewritten to spell out the new contract and the -1 semantics. NativeStorageDevice.Initialize remarks updated. Verified on Linux net10.0 Release: * 599 hlog tests (491 passed + 108 skipped) * 305 recovery tests * 144 + 155 + 127 + 346 = 772 other Tsavorite + Garnet RespTests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite: native devices honor omitSegmentIdFromFilename; O_TMPFILE probe (no race) Two related fixes on top of the unified Initialize contract: 1) NativeStorageDevice now supports omitSegmentIdFromFilename ───────────────────────────────────────────────────────── Previously Native rejected the omit flag because the C++ shim hard-coded the '.<segmentId>' suffix in three places in file_system_disk.h. This made the IDevice contract asymmetric (managed devices honored omit, Native didn't). Fix by threading the bool through the entire C++/C ABI: C++ (libs/storage/Tsavorite/cc/src/device/): * FileSystemSegmentBundle: new bool omit_segment_id_; both ctors accept it and use a new segment_path(idx) helper that returns just filename_ when set, otherwise filename_ + '.' + std::to_string(idx). Used at all three locations that previously hard-coded the suffix. * FileSystemSegmentedFile: new bool omit_segment_id_ (const) wired through ctor and propagated to bundles allocated by OpenSegment. * NativeDeviceImpl: new bool omit_segment_id constructor param; recorded as omit_segment_id_ member. ValidateRecoveredSegments short-circuits in omit mode (single bare-named file, segment-size mismatch check is meaningless when there's no .<id> suffix to scan for). * native_device_wrapper.cc / NativeDevice_CreateWithBackend: new trailing 'bool omit_segment_id' parameter. ABI BUMP — managed wrapper updated to match; Linux .so rebuilt and committed at libs/storage/Tsavorite/cs/src/core/Device/runtimes/linux-x64/native/ libnative_device.so. **Windows DLL must be rebuilt by user** with cmake -G 'Visual Studio 17 2022' -A x64 -T v143,spectre=true. C# (libs/storage/Tsavorite/cs/src/core/Device/): * NativeStorageDevice P/Invoke signature updated. * NativeStorageDevice.Initialize removes the 'always rejects omit' guard. It now accepts omit:true together with segmentSize = -1 and forwards to native; rejects omit:true together with a positive segmentSize with a clear error message (multiple segments would collapse onto the same on-disk path and clobber each other). Tests (DeviceTests.cs): * IDevice_Initialize_OmitSegmentIdFromFilename_BareFileName(kind) × 3: writes via Initialize(-1, omit:true) and asserts the on-disk file is the bare basename (no .0 suffix). Replaces the native-only 'throws' test from the previous commit. * IDevice_Initialize_OmitSegmentIdFromFilename_WithoutMinusOne_Throws(kind) × 3: enforces the no-positive-size-with-omit invariant on every kind. 2) IsDirectIOSupported uses O_TMPFILE (race-free probe) ───────────────────────────────────────────────────── The previous probe in libs/storage/Tsavorite/cs/src/core/Device/ LinuxFileExtensions.cs created a hidden '.tsavorite-odirect-probe-<pid>- <guid>' file in the device's directory, then File.Delete'd it in a silent-catch finally. Multiple concurrent commits (one device per Get()) ran probes simultaneously; concurrent ListContents calls from CommitRecordBoundedGrowthTest would observe the probe file during its brief lifetime, and DefaultCheckpointNamingScheme.CommitNumber would then throw FormatException on long.Parse(''). 18/20 baseline failure rate. Switching from create+unlink to open(directory, O_TMPFILE | O_RDWR | O_DIRECT) tells the kernel to allocate an anonymous inode in the directory's filesystem with NO directory entry. The probe inode is invisible to readdir/getdents regardless of timing; concurrent ListContents cannot observe it. Freed on close. Linux >= 3.11 + ext4/xfs/tmpfs/btrfs all support it. If O_TMPFILE itself fails (EOPNOTSUPP on some filesystem) we conservatively report 'no O_DIRECT' so the device falls back to the page-cache path — no named-file fallback because that's the bug we're fixing. Reverts the dotfile filter in LocalStorageNamedDeviceFactory.ListContents added by the previous commit; the underlying race is now eliminated at the kernel level so the workaround is unnecessary. 20/20 LogFastCommitTests runs pass after the change (was 2/20 on baseline, 3/20 on the previous unlink-after-open attempt which still raced). Verified on Linux net10.0 Release: * 612 hlog tests (504 passed + 108 skipped) * 305 recovery tests * 62 device tests (IDevice contract + NativeStorageDevice-specific) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * KV.benchmark: add disk-IO thread-scale sweep recipe to cookbook Captures the 100M-key disk-bound thread-scale experiment from the optimize-device branch sweep so it can be reproduced verbatim in the future: - 100M × 100B records (12.8 GB on disk) - 16 MB log so ~0.125 % of dataset is in memory and almost every read is a 4 KB random disk fetch - 8 load threads, run-threads sweep 1,2,4,8,16,32 at 15s each - One row per backend (RandomAccess / native+libaio / native+uring) Added a short note after the table explaining what to compare against (the disk's fio ceiling at 4K-aligned QD=64-per-job), the expected ~2 min wall-clock per device, and the observed per-backend plateau characteristics so the next operator knows what 'good' looks like. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Device.benchmark + KV.benchmark: robustness & flag improvements Device.benchmark fixes (previously reported throughput could be 2x inflated): - Throughput counter now tallies successful completions only. Before, every ReadAsync call was counted as success even when the kernel returned EAGAIN (Status::IOError=4 — flooded libaio io_context ring). Under --throttle-limit 0 with high QD, ~40% of "ops" were errored requests. - Per-error-code histogram printed at end of run; no per-error Console.WriteLine (was emitting millions of serial writes/run, both falsifying numbers and slowing the real path). - DEBUG data validation skipped on errored ops (was reading garbage from the destination buffer on EAGAIN paths and reporting spurious "Data mismatch"). - --throttle-limit help text documents the libaio kernel ring trap (128 slots wide; high QD + no throttle floods it) and recommends --throttle-limit 128 (also the io_uring SQ depth this build uses). - --io-backend flag added (libaio / uring / default) so the existing Linux Native path can be exercised against either backend. Unknown values are rejected with an actionable message at startup instead of silently falling back to default. - --completion-threads is now wired through to the Linux Native ctor (was hardcoded to 1). - --file-size widened to long (was int; --file-size > 2GB threw a parse error). KV.benchmark + Devices.cs: clarified XML/help text for the existing --device-completion-threads / numCompletionThreads parameter to describe the current behavior (multiple drainer threads share one kernel io_context / io_uring per device). No behavior change for KV.benchmark or core Tsavorite Devices.cs API beyond the Device.benchmark surface and clarified docs. * Tsavorite Native: shard io_uring per completion thread; new C ABI Adds N independent kernel io_contexts (libaio) / io_urings (uring) per NativeStorageDevice, with completion threads bound 1:1 to contexts. libaio internally always uses 1 context (sharding empirically gave nothing — kernel mutex efficient at all tested loads), but the sharded ABI surface is kept across both backends so the templated NativeDeviceImpl<HandlerT> doesn't need a fork. C ABI changes (libnative_device.so): - NativeDevice_CreateWithBackend signature bumped: trailing int32 num_io_contexts. - New exports: NativeDevice_QueueRunFor(device, ctx_idx, timeout_secs), NativeDevice_NumIoContexts(device). - Legacy NativeDevice_QueueRun kept; under uring sharding it scans all rings (back-compat for any single-thread drainer). C# (NativeStorageDevice): - Synchronous ABI probe at Initialize() that converts EntryPointNotFoundException into a clear TsavoriteException listing the missing exports and how to rebuild — guards against a stale .so silently hanging Dispose's drain loop. - Probe is intentionally gated by the QueueRun branch so it runs only on backends that actually use the new symbols at runtime: Linux Native (libaio / uring) where QueueRun returns >= 0, NOT on Windows IOCP where the ThreadPoolIoHandler returns -1 by design. This means a stale Windows DLL keeps working unchanged because it never calls the sharded exports. cdecl/x64 ABI silently tolerates the new trailing num_io_contexts arg on NativeDevice_CreateWithBackend. - Completion threads bound 1:1 via QueueRunFor(ctxIdx) (no closure capture bug — ctxIdx is captured per-iteration into a local). - numCompletionThreads is the user-facing knob; native side decides how many contexts to actually create (libaio: always 1; uring: honours the request). Empirical justification (Device.benchmark, NVMe, 4K random reads, batch=4096, throt=512, fio ceiling 749K, NUMA0-pinned): libaio CT=1 (always 1 ctx + 1 drainer): 755K ops/sec (t=32) uring CT=1 (1 ring + 1 drainer): 357K ops/sec ← SpinLock-bound uring CT=1 ring + N drainers (regresses): 357K → 274K ← cq_lock contention uring CT=4 (4 rings + 4 drainers): 745K ops/sec uring CT=8 (8 rings + 8 drainers): 758K ops/sec ← hardware ceiling Both backends now reach the hardware NVMe ceiling. uring requires sharding (the user-space SpinLock around io_uring_get_sqe + prep + submit is the real cap). libaio doesn't need sharding (kernel io_context mutex already efficient at all tested loads). Files: - file_linux.h : UringIoHandler sharded (vector<io_uring*>, per-ring sq_lock + cq_lock, atomic round-robin pick_ring). QueueIoHandler unchanged on the data plane (single io_context_t) but exposes the same num_contexts()/TryCompleteFor/QueueRunFor surface as inline stubs for ABI symmetry. - file_linux.cc : new UringIoHandler impls; QueueIoHandler unchanged. - file_windows.h: stub overloads (num_contexts()=1, QueueRunFor=-1, 2-arg ctor) so the templated NativeDeviceImpl compiles unchanged. - native_device.h, native_device_wrapper.cc: ABI plumbing as above. - NativeStorageDevice.cs: ABI probe + per-context drain workers. - runtimes/linux-x64/native/libnative_device.so: rebuilt with sharding. * Device.benchmark: add cookbook README showing how to saturate ~750K NVMe IOPS Captures the verified copy-paste recipe for both Linux Native backends (libaio and io_uring) to hit the hardware ceiling on a Dell P5600-class NVMe, alongside a flag reference, output-schema explanation, and troubleshooting table. Headline recipes verified end-to-end on the reference setup: libaio --completion-threads 1 --threads 16 --throttle-limit 512 → 743K ops/sec uring --completion-threads 8 --threads 16 --throttle-limit 512 → 738K ops/sec (Both within 2 % of the table values in the README; zero kernel-side errors.) Key facts documented: - libaio always uses one io_context in this build regardless of --completion-threads (sharding empirically gave nothing; the hint is ignored). Pass 1 explicitly so scripts are self-describing. - io_uring needs sharded rings (CT >= 4) to escape the per-ring user-space SpinLock cap around io_uring_get_sqe + prep + submit; CT=8 is the safe peak. - --throttle-limit must be set to at least the per-ring/per-context depth (128 in this build for both backends). --throttle-limit 0 floods the kernel ring and the benchmark correctly surfaces Status::IOError=4 in the per-code histogram rather than tallying errored ops as throughput. - --file-size must be a multiple of 1024 × --sector-size (fill phase uses a 1024-sector temp buffer). Plus a section comparing Device.benchmark vs KV.benchmark to direct readers to the right tool: Device.benchmark for IO-layer ceiling validation (saturates NVMe), KV.benchmark for full-path throughput (currently caps ~30 % below the IO ceiling on the upper-layer pending-read path — see KV.benchmark README for that side of the story). Also adds a one-paragraph pointer in benchmark/README.md so the new README is discoverable from the top-level benchmarks listing. * Tsavorite Native: bounded sched_yield retry on transient kernel-ring full ScheduleOperation in both QueueFile (libaio) and UringFile (io_uring) now retries the kernel-side submission on the transient back-pressure signal (libaio: `io_submit == 0`; uring: `io_uring_get_sqe == nullptr`) up to kMaxSubmitRetries = 8 attempts, each separated by a sched_yield(). Permanent errors (libaio io_submit < 0, uring io_uring_submit < 0) are NEVER retried — they surface immediately as Status::IOError. Motivation: the upper-layer throttle gate in AllocatorBase.AsyncGetFromDisk is a racy test-then-increment (Throttle() reads numPending non-atomically, then ReadAsync does Interlocked.Increment). With N concurrent submitters all passing the gate at numPending == ThrottleLimit, in-flight can spike to ThrottleLimit + N momentarily, exceeding the 128-slot per-context / per-ring kernel ring depth when N > 8 (which is normal for Garnet under heavy disk-bound load). Pre-fix, the kernel rejects the overshoot submissions with EAGAIN, which Tsavorite handled by re-routing through the full AllocatorBase pending-read retry loop (correct but expensive: a full round-trip per IOError). Post-fix, the burst is absorbed locally by a handful of sched_yields and never surfaces to the upper layer. Why sched_yield + bounded retries is the right shape: on a 750K-IOPS NVMe the kernel ring drains a slot every ~1.3 µs and sched_yield is typically 1-10 µs on Linux, so 8 retries (worst-case ~40-80 µs window) is more than enough to absorb the typical 24-slot overshoot from a 32-thread burst. For genuine sustained overload (application submission rate exceeds device IOPS for seconds), the retries exhaust and Status::IOError surfaces — which is the correct signal for the caller to apply back-pressure. Implementation notes: - libaio: simple loop around io_submit. No lock to release; io_submit is kernel-thread-safe per io_context, so concurrent submitters serialise inside the kernel. - uring: must release sq_lock around sched_yield. Holding a SpinLock across a syscall would stall every other submitter on the same ring. Only the get_sqe == nullptr path is retried; if get_sqe succeeded we've already "consumed" an SQE slot in the user-side bookkeeping and re-issuing via get_sqe+prep on retry would corrupt the ring (we'd hold two SQEs for one logical op). For SQPOLL-disabled rings (our setup) io_uring_submit returns 1 in steady state — a non-1 there is an unrecoverable kernel-side error and surfaces immediately. Verified end-to-end (Device.benchmark, NVMe, 4K random reads, batch=4096, NUMA0-pinned, --throttle-limit 120 to match production NativeStorageDevice default): libaio CT=1: t=8 / 16 / 32 → 626K / 631K / 627K ok/sec, 0 err uring CT=8: t=8 / 16 / 32 → 622K / 625K / 626K ok/sec, 0 err At extreme intentional-overload settings (--throttle-limit 4096, t=32) errors still appear — confirming the retry budget correctly exhausts when the application is genuinely outpacing the device: libaio CT=1, t=32, throt=4096: 754K ok, 14.7M code4 err (~5% of submits) uring CT=8, t=32, throt=4096: 745K ok, 0 err (uring still produces 0 err under same overload because 8 rings × 128 = 1024 SQ slots is large enough that even gross over-submission fits within the retry budget per ring.) * Tsavorite Native: PR review fixes (3-model code review pass) BLOCKER fixes: - NativeStorageDevice.Dispose UAF race. Previously NativeDevice_Destroy(nativeDevice) ran before nativeDevice was nulled, so a concurrent guard-bypassed P/Invoke could observe a non-zero handle that points to freed memory. Fix: Interlocked.Exchange atomically captures-and-nulls the handle; destroy runs on the captured pointer. EnsureReadyOrSilent now checks disposedFlag first. HIGH fixes: - UringFile::ScheduleOperation SQE leak on transient io_uring_submit failure. After a successful get_sqe + prep, the SQE is committed to the user-side SQ ring; a -EAGAIN/-EBUSY return from io_uring_submit left the slot permanently occupied with no kernel iocb, eventually starving get_sqe forever. Fix: retry io_uring_submit (without re-preparing) up to kMaxSubmitRetries on transient negatives, with sched_yield (and sq_lock released) between attempts. - UringIoHandler::Init partial-init leak. If new SpinLock() threw after io_uring_queue_init succeeded for ring i, the already-initialized ring leaked (the class dtor doesn't run on partial construction). Fix: use std::unique_ptr RAII holders during construction; release into the member vectors only after all allocations succeed. POLISH fixes: - NativeStorageDevice.Initialize tail-throw cleanup. If base.Initialize threw after the native device and completion threads were created, both leaked. Now wrapped in try/catch that cancels token, joins threads, and destroys the native device. - UringIoHandler rule-of-5 hygiene: explicitly deleted copy ctor, copy-assign, and move-assign so the implicit shallow copies (which would double-delete the raw owning pointers) cannot be generated. - DispatchUringCqe: added static_assert(is_trivially_destructible<IoCallbackContext>) so a future non-trivial member fails the build instead of silently leaking. - NativeDeviceImpl::num_io_contexts: removed unnecessary const_cast (the underlying num_contexts() is already const on both backends). - Removed duplicate XML <summary> on NativeStorageDevice.Dispose. - FileSystemDisk dead-code ctor: passed the now-required 5th arg to FileSystemSegmentedFile so the file compiles if anyone instantiates it. Comment hygiene sweep (per explicit review-rule #4 "comments should not refer to design thought processes"): - Removed embedded benchmark results, hardware-specific throughput numbers, and historical narrative from class/method documentation in file_linux.{h,cc}, native_device.h, native_device_wrapper.cc, NativeStorageDevice.cs. - Kept WHAT each method does and the invariants it enforces; moved WHY this approach was chosen out of source comments (the commit log is the appropriate place for that context). - Net: -162 lines across 7 files, no behavior change from the trim itself. Verified post-fix performance unchanged (Device.benchmark, NVMe, 4K random reads, batch=4096, NUMA0-pinned): libaio CT=1 t=16 throt=512: 746K ok/sec, 0 err (hardware ceiling) uring CT=8 t=16 throt=512: 743K ok/sec, 0 err (hardware ceiling) libaio CT=1 t=32 throt=120: 636K ok/sec, 0 err (production default) uring CT=8 t=32 throt=120: 618K ok/sec, 0 err (production default) * Tsavorite Native: make Initialize idempotent via lazy native-handle creation NativeStorageDevice.Initialize used to perform all the heavy work (native device creation, completion-thread spawn, ABI / segment-size / sector-size cross-checks) eagerly and threw "called more than once" on a second call. The other IDevice implementations (LocalStorageDevice, RandomAccessLocalStorageDevice, ManagedLocalStorageDevice) all inherit a metadata-only StorageDeviceBase.Initialize that simply overwrites segmentSize / segmentSizeBits / mask fields and is silently idempotent. They open their per-segment OS handles lazily inside the IO methods. This contract mismatch broke any caller that invokes Initialize twice on the same NSD instance. The canonical case is LocalStorageNamedDeviceFactory.Get(), which calls Initialize(-1L) as a defensive pre-init so consumers can't forget; the consumer (snapshot checkpoint state machine SnapshotCheckpointSMTask, cluster checkpoint streaming TsavoriteCheckpointReader.CreateCheckpointDevice) then calls Initialize(actualSegmentSize). Under the old NSD that throws; under the new NSD it works the same way the other backends do. Implementation: - NSD.Initialize is now metadata-only — delegates to base.Initialize. Pre-flight argument validation (segmentSize power-of-two, sector-size floor, omitSegmentIdFromFilename) is preserved. - New EnsureNativeDeviceCreated() does the heavy work, lazily, on first IO. Reads the latest base.segmentSize and base.OmitSegmentIdFromFileName so whichever Initialize call ran most recently wins. - Thread-safe via double-checked locking on a new nativeCreateLock. The publish of nativeDevice uses Volatile.Write so a second observer of nativeDevice != IntPtr.Zero is guaranteed to see a fully-initialised handle with completion threads already running. - Dispose now also takes nativeCreateLock around the cancel-join-destroy sequence so it cannot race with a concurrent EnsureNativeDeviceCreated (which would otherwise leak a freshly-published native handle and its completion threads). - IO entry points (ReadAsync, WriteAsync) call EnsureNativeDeviceCreated() before submission. Bookkeeping entry points (Reset, TryComplete, GetFileSize, RemoveSegment) no-op when the native handle has not been created yet, matching the semantics of the other backends (Reset on a device with no open handles is a no-op). Verified against the full unit-test sweep with Native forced as the default device (the GetDefaultDeviceType hack is local-only and not in this commit): Tsavorite.test: 206 / 206 (was 204 / 206 pre-fix) Garnet.test: 789 / 789 (was 110 / 792 pre-fix; 681 were blocked on Initialize-twice) Garnet.test.acl: 425 / 425 Garnet.test.collections: 746 / 746 Garnet.test.complexstring: 386 / 386 Garnet.test.rangeindex: 62 / 62 Garnet.test.vectorset: 42 / 42 No change to behavior for callers that invoke Initialize once with the real segment size, which is what every production code path already does. * Tsavorite Native: probe GetFileSize/RemoveSegment without forcing native handle GetFileSize and RemoveSegment must report the on-disk state regardless of whether IO has flowed through the device, matching LocalStorageDevice and RandomAccessLocalStorageDevice semantics. Before this fix, both no-op'd when no native handle had been created — which silently truncated the cluster manager's recovery decision because ClusterManager.cs:79 and ReplicationManager.cs:160 call `device.GetFileSize(0) > 0` to decide whether to recover persisted cluster config / replication history. With Native, a restarted node would always "Initialize new node instance config" instead of recovering, get a fresh node ID, and fail every replication-resume test (e.g. ClusterSRNoCheckpointRestartSecondary which restarts a replica and then waits for AOF sync to catch up). Changes: * GetFileSize now falls back to FileInfo when no native handle exists — same shape as RandomAccessLocalStorageDevice.GetFileSize (open-on-demand) but without paying io_uring/libaio setup cost just to stat a file. * RemoveSegment now falls back to File.Delete when no native handle exists — same shape as LocalStorageDevice / RandomAccessLocalStorageDevice (best-effort unlink, swallows ENOENT). * Per IDevice contract enforced in 889def4 ("Tsavorite IDevice: unify Initialize contract — required for all devices, -1 = unbounded single segment"), ReadAsync / WriteAsync now call EnsureInitialized() before EnsureNativeDeviceCreated() so the IDevice_*BeforeInitialize_Throws hardening tests get the same InvalidOperationException shape from Native that they get from the other devices. * Two device tests updated to match the lazy-Initialize contract that was introduced in commit f4e3044 ("Tsavorite Native: make Initialize idempotent via lazy native-handle creation"): - NativeStorageDevice_InitializeTwice_Throws → _Idempotent: idempotent Initialize matches the LSD/RA contract used by LocalStorageNamedDeviceFactory.Get + consumer re-init pattern. - NativeStorageDevice_Recovery_LargerExistingSegment_DetectsMismatch: the C++ ValidateRecoveredSegments check now fires on first IO (when EnsureNativeDeviceCreated runs), not at Initialize time, so the test asserts on a ReadAsync rather than Initialize. Verified on Linux x64 / .NET 10: * libs/storage/Tsavorite/cs/test/test.hlog: all IDevice + NativeStorageDevice tests pass (62/62) with Native default * test/cluster/Garnet.test.cluster.replication: all 4 ClusterSRNoCheckpointRestartSecondary variants pass with Native default (regression-test for the recovery path) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: wake completion drainer on Dispose via no-op IO Without this fix, NSD.Dispose() can stall up to CompletionWorkerTimeoutSecs (1s) per io_context because the completion-drainer thread is blocked in io_getevents / io_uring_wait_cqe_timeout waiting for events that will never come (the IO drain phase has already brought numPending to 0). The Thread.Join following completionThreadToken.Cancel() then has to wait for the next QueueRunFor timeout to fire so the thread can observe cancellation and exit. This was visible as exactly-1.0s gaps in cluster replication recovery traces: checkpoint metadata reads / writes that each create+dispose a fresh NSD spent ~1s in Dispose, multiplying across the ~5–10 devices created per checkpoint into multi-second stalls. ClusterReplicaSyncTimeoutTest (replicaSyncTimeout=1s) and MultiDatabaseSaveRecoverByDbIdTest(True) (2s LASTSAVE poll window) failed because of this; the actual I/O on Native is microseconds, not seconds. Fix: post a synthetic wake-up event on each io_context when Dispose runs. * libaio: submit a 0-byte read on a /dev/null fd opened in the handler ctor. /dev/null completes immediately and does not require O_DIRECT alignment, so the wake-up does not interfere with the real segment files. * io_uring: submit io_uring_prep_nop with user_data = nullptr; the drain loop recognises nullptr as a wake-up sentinel and skips dispatch. * Windows ThreadPoolIoHandler has no dedicated drainer (callbacks fire on threadpool threads), so its Wake is a no-op stub returning 0. The completion thread wakes from its blocking syscall almost immediately, observes the cancellation token on its next loop iteration, and exits. No extra idle work, no polling, no shortened timeout. * NSD.Dispose latency: 1025ms worst case -> ~20-30ms (microbenchmark). * ClusterReplicaSyncTimeoutTest with Native: ~22-25s (fail) -> ~3s (pass). * MultiDatabaseSaveRecoverByDbIdTest(True) with Native: timeout (fail) -> ~6s (pass). * Idle drainer syscall rate is unchanged (1/s/context). C ABI changes (additive — old exports preserved): * NativeDevice_WakeCompletionWorker(device, ctx_idx). * INativeDevice::Wake; QueueIoHandler::Wake, UringIoHandler::Wake, ThreadPoolIoHandler::Wake stub. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite IDevice: make Initialize() optional — ctor defaults are valid for IO Background: commit 889def4 ("unify Initialize contract — required for all devices") added an EnsureInitialized() guard that threw InvalidOperationException at every IO entry point if Initialize() had not been called first. This was redundant: the ctor already establishes segmentSize=-1 / segmentSizeBits=64 / segmentSizeMask=~0UL, which is functionally identical to having called Initialize(-1) — every absolute address right-shifts to segment 0, producing unbounded single-segment routing. The mandatory-Initialize contract was the root cause of the entire factory-pre-init + NSD lazy-creation saga: LocalStorageNamedDeviceFactory.Get was forced to call device.Initialize(-1L) defensively just to satisfy the contract, which broke NativeStorageDevice (its single-shot Initialize then asserted on the consumer's follow-up Initialize(realSize)). Recent commits f4e3044 + 0535da0 papered over this with lazy native-handle creation; this commit removes the root cause. Changes: * StorageDeviceBase: remove the 'initialized' flag, EnsureInitialized() helper, and ThrowNotInitialized() method. Initialize() is now purely a *configuration* call to override the ctor defaults (set a non-default segment size, opt into OmitSegmentIdFromFileName). The ctor doc explicitly states that callers may issue IO immediately after construction. * All concrete devices: remove the EnsureInitialized() calls at the top of ReadAsync / WriteAsync / TruncateUntilSegmentAsync / RemoveSegment (libaio, io_uring, RA, ManagedLocal, LocalMemory, Null, Tiered, Sharded, Azure). * LocalStorageNamedDeviceFactory.Get: drop the defensive device.Initialize(-1L); the ctor defaults match what that call did anyway. * NSD's recent EnsureInitialized() additions to ReadAsync/WriteAsync (introduced in 0535da0 only to satisfy the hardening test) are also removed by the sweep. * ComponentRecoveryTests.cs: drop 3 redundant Initialize(-1L) calls. * test.hlog DeviceTests: rename and repurpose IDevice_*BeforeInitialize_Throws to IDevice_*BeforeInitialize_UsesCtorDefaults — the new test demonstrates that WriteAsync/ReadAsync on a freshly-constructed device (no Initialize call) works correctly using the unbounded single-segment defaults, across Native / RA / ManagedLocal. TestUtils.cs:165 still calls device.Initialize() — that path is conditional on the caller wanting OmitSegmentIdFromFileName=true, which IS only settable via Initialize (it is not a ctor parameter), so the call is genuinely needed there. Verified on Linux x64 / .NET 10: * libs/storage/Tsavorite/cs/test/test.hlog (IDevice + NativeStorageDevice tests): 62/62 pass. * libs/storage/Tsavorite/cs/test/test.recovery (ComponentRecovery tests): 4/4 pass. * Full Garnet.test, Garnet.test.cluster, Garnet.test.acl, Garnet.test.collections, Garnet.test.extensions, Garnet.test.scripting, Garnet.test.complexstring, Tsavorite IDevice+NSD: pass at the same rates as before this commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native + IDevice: refresh comments to reflect final design Sweep the PR for comments that referenced earlier design choices as they evolved during development, and rewrite them to describe the steady-state contract directly without historical baggage. * IDevice.Initialize: replace the "Must be called exactly once... before any IO entry point... uninitialized device throws InvalidOperationException" doc with the actual contract: Initialize is purely an opt-in configuration step to override the ctor defaults (which are equivalent to Initialize(-1)); callers may issue IO immediately after construction. * NativeStorageDevice ctor doc: rewrite to describe the as-shipped lazy creation flow (configuration captured at ctor, native handle created on first IO via EnsureNativeDeviceCreated) rather than the stale "Native device creation is DEFERRED until Initialize... every IO entry point throws InvalidOperationException" framing. * NativeStorageDevice.Initialize doc: drop the misleading "Creates the underlying native device with the requested segment size" lead-in (which hasn't been true since the lazy-creation refactor); replace the "factory pre-init" example (factory no longer pre-inits) with a steady-state description of when repeat Initialize calls are honoured. * NativeStorageDevice.EnsureNativeDeviceCreated doc: replace "Throws if Initialize has not been called" (now uses ctor defaults if no Initialize) with "Throws if the device has been disposed or if the native shim rejects the configuration". * NativeStorageDevice.EnsureReadyOrSilent doc: drop the "does not throw on 'not initialized yet'" qualification. * NativeStorageDevice.GetSectorSize doc: re-point the "cross-check" link from Initialize to EnsureNativeDeviceCreated (which is where it actually happens). * NativeStorageDevice.Dispose doc + body: bound the worst-case shutdown stall by the longest in-flight user callback (not CompletionWorkerTimeoutSecs) since wake-up uses NativeDevice_WakeCompletionWorker; rewrite the inline Dispose comment so it documents the steady-state design rather than what it improved over. * NativeStorageDevice nativeSegmentSizeBytes / UnboundedNativeSegmentSizeBytes field doc: clarify that the value is populated by EnsureNativeDeviceCreated (not Initialize) and that the default is reached without calling Initialize. * NativeStorageDevice_InitializeTwice_Idempotent test: drop the now-stale "factory pre-init... consumer re-initializes" rationale; describe the idempotent contract directly. * NativeStorageDevice_DisposeBeforeInitialize_IsNoOp test: drop the "Phase 6" reference and reword in terms of the steady-state lazy-creation contract. * SimulatedFlakyDevice.Initialize: replace the "so its EnsureInitialized() guard passes when our IO methods delegate to it" comment (the guard no longer exists) with a description of why both devices need matching geometry. * LinuxFileExtensions.OpenDirect dsync param doc: drop the "previously asked for it" wording — the WriteThrough callsites still pass it; describe the parameter as an opt-in for WriteThrough-equivalent semantics. * Doc-cref bookkeeping: change <see cref="base.segmentSize"/> (illegal cref for inherited fields) to <c>base.segmentSize</c> code spans. No behaviour change. Build clean on Garnet.slnx and Tsavorite.slnx; dotnet format --verify-no-changes clean on both. IDevice + NSD device tests all pass (62/62). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: ship libnative_device.so without liburing dependency Background: the shipped libnative_device.so was built with -DUSE_URING=ON, so it had a hard DT_NEEDED entry for liburing.so.2. Loading the .so on a host without liburing2 installed (e.g. a GitHub Actions ubuntu-latest runner, or an end-user box where only libaio is in the base image) failed with: System.DllNotFoundException: ... liburing.so.2: cannot open shared object file: No such file or directory …even for callers that only ever requested the libaio backend, because the dynamic linker resolves NEEDED libraries at load time regardless of which exported symbols the caller goes on to invoke. Rebuild the prebuilt with -DUSE_URING=OFF so the shipped .so links only libaio. Most Linux distributions ship libaio in the base system, so the prebuilt now loads without any additional setup. The io_uring backend becomes a build-time opt-in: callers that want it install liburing-dev and rebuild with -DUSE_URING=ON. The C# layer already surfaces a clear TsavoriteException for callers that request Uring against a USE_URING=OFF build ("Requested IO backend 'Uring' is not available in the loaded native_device library… Rebuild the native library with -DUSE_URING=ON and install liburing-dev to enable io_uring."). Build fix: file_linux.h now includes <fcntl.h> directly (for the ::open() / O_RDONLY usage in QueueIoHandler::OpenWakeFd()). Previously these were pulled in transitively through <liburing.h>, which is now gated behind #ifdef FASTER_URING. Dockerfile updates: drop liburing2 / liburing from the runtime install list in all 5 Dockerfiles (default, .ubuntu, .alpine, .azurelinux, .chiseled). Comments left for users that rebuild with USE_URING=ON. README updates: rewrite the "Runtime dependencies" section to describe the new default (libaio only). Replace the "Disabling io_uring (optional)" section with "Enabling io_uring (optional)". Verified on Linux x64 / .NET 10: libaio default works (62/62 IDevice + NativeStorageDevice tests pass); ldd confirms only libaio.so.1t64 is in NEEDED. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: ship two .so flavors — uring-enabled + libaio-only fallback Single-shipping libnative_device.so created a deployment dilemma: build with USE_URING=ON and end-users without liburing get DllNotFoundException at load time; build with USE_URING=OFF and the io_uring backend stops working even on hosts that DO have liburing installed (which is the case where uring matters for perf — modern NVMe at >1M IOPS benefits noticeably from uring over libaio). Ship both flavors instead: * libnative_device.so — built with USE_URING=ON; DT_NEEDED on libaio AND liburing. Exposes both Libaio and Uring backends. * libnative_device_libaio.so — built with USE_URING=OFF; DT_NEEDED on libaio only. Exposes the Libaio backend. NativeStorageDevice's DllImportResolver tries the uring-enabled binary first; on DllNotFoundException matching 'liburing.so.2: cannot open' it falls back to the libaio-only binary. The Libaio backend therefore always works out of the box on any Linux distribution that ships libaio (essentially all of them). liburing is opt-in: hosts that install it get the Uring backend with zero runtime overhead vs Libaio (direct calls, no function-pointer indirection — we deliberately rejected the dlopen approach so the future-default uring path stays optimal). If a caller explicitly selects IoBackend.Uring on a host without liburing, the construction-time error message now points at the install command per distro ('apt-get install -y liburing2', 'dnf install -y liburing', 'apk add liburing') instead of telling the user to rebuild the .so with -DUSE_URING=ON. We never silently downgrade Uring to Libaio. Changes: * NativeStorageDevice.cs: new LibaioFallbackLibraryPath; ImportResolver catches DllNotFoundException for liburing.so.2 and falls back to the libaio-only .so. ResolveNativeLibraryPath now takes the path as a parameter so it can resolve either flavor. * NativeStorageDevice.cs: rewrite the 'backend not available' exception message — point at install commands (the actual remediation) not rebuild. * Tsavorite.core.csproj: add libnative_device_libaio.so as a second ContentWithTargetPath asset so both .so files are copied to the output directory and packed into the NuGet runtime payload. * runtimes/linux-x64/native/libnative_device.so — REPLACED with USE_URING=ON build (DT_NEEDED libaio + liburing). 2.3 MB. * runtimes/linux-x64/native/libnative_device_libaio.so — NEW, USE_URING=OFF build (DT_NEEDED libaio only). 1.6 MB. * Dockerfile, Dockerfile.ubuntu, Dockerfile.alpine, Dockerfile.azurelinux, Dockerfile.chiseled: re-add liburing2 / liburing to the runtime installs so docker users get the io_uring backend out of the box (the libaio-only fallback would otherwise leave Uring unusable inside containers). * cc/README.md: rewrite the 'Runtime dependencies' and build sections to describe the two-flavor layout, drop the stale 'Enabling io_uring' section, and document the prebuilt rebuild workflow. Verified end-to-end: * Both backends saturate the Dell P5600 NVMe at ~743K random read IOPS in benchmark/Device.benchmark (matches the pre-change reference). * 62/62 IDevice + NativeStorageDevice tests pass. * dotnet format clean on both Garnet.slnx and Tsavorite.slnx. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native (Windows): add init_errno()/initialized() stubs to ThreadPoolIoHandler NativeDeviceImpl's constructor (native_device.h:100-101) gates on handler_.init_errno() to surface an actionable error message when the underlying IO handler failed to initialize (e.g., libaio io_setup() failed with EMFILE / ENOMEM, or io_uring_queue_init() failed). The Linux handlers QueueIoHandler and UringIoHandler both expose this API; ThreadPoolIoHandler (Windows) did not, so MSVC failed to instantiate NativeDeviceImpl<ThreadPoolIoHandler> with: error C2039: 'init_errno': is not a member of 'FASTER::environment::ThreadPoolIoHandler' Add init_errno() and initialized() stubs that return 0 / true unconditionally — the Windows ThreadPool API does not have a separable init step that can fail in the same way the Linux io_setup / io_uring_queue_init paths can (threadpool creation failures propagate via threadpool_'s ctor, not via a later 'check this' field on the handler), so the stubs are semantically correct. NativeDeviceImpl then falls through to the log_.Open(&handler_) path which is where Windows-specific errors (missing directory, permission denied, etc.) actually surface. Linux unaffected: rebuilt build/Release-uring cleanly after this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native tests: stop skipping NSD tests on Windows The Native tests in DeviceTests.cs had blanket 'NativeStorageDevice is Linux-only' Assert.Ignore guards that dated from when NSD's C++ shim was Linux-only. The shim is built on Windows too (native_device.dll via the ThreadPool / IOCP backend in file_windows.cc), so directly constructing 'new NativeStorageDevice(...)' works on Windows. The blanket guards were silently dropping ~15 NSD test cases on Windows CI. Drop the guards so the tests exercise the Windows C++ shim. The legitimate Linux-only guard on IDevice_PermissionDeniedAtFirstWrite_CallbackGetsError (chmod-based; chmod has no Windows analogue) is preserved. Important: end-user device routing is UNCHANGED. Devices.CreateLogDevice(DeviceType.Native) on Windows still returns LocalStorageDevice (managed Windows IOCP), not NativeStorageDevice — that routing happens in Devices.cs and was not touched. These tests directly instantiate the NSD class for shim-coverage purposes only; they do not affect what end users get from the default device factory. Linux: 62/62 pass after this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: rebuild win-x64 native_device.dll for the latest C++ source Rebuild the shipped Windows prebuilt from the current native_device source so the latest fixes (Initialize idempotence, WakeCompletionWorker, etc.) are reflected in the win-x64 DLL. USE_URING is a no-op on Windows; the DLL only exposes the Default (IOCP) backend, so there is no equivalent of the libnative_device_libaio.so fallback on this platform. End-user device routing is unchanged: Devices.CreateLogDevice(DeviceType.Native) on Windows still returns LocalStorageDevice (managed Windows IOCP). This DLL is exercised by direct 'new NativeStorageDevice(...)' construction (see Tsavorite.test.hlog DeviceTests — 59/59 Native + IDevice tests pass on Windows after this rebuild). Built with: Visual Studio 17 2022, MSVC v143, x64, Release configuration, Spectre-mitigated CRT. * Tsavorite Native: address GPT-5.5 PR review findings Fixes two correctness issues caught by an automated code review of the optimize-device PR. ### io_uring SQE leak on submit failure In UringFile::ScheduleOperation, io_uring_get_sqe() advances the user-side sqe_tail before io_uring_submit() is called. If submit fails after retries (-EAGAIN/-EBUSY exhausted, or any other negative), the old code released the lock and returned IOError without doing anything about the still-pending SQE. user_data on that SQE pointed at the io_context unique_ptr that was about to be freed by the guards unwinding, so the next successful submit on the same ring would consume the stale SQE and the QueueRunFor drain loop would dispatch a callback against freed memory — a clear use-after-free. Fix: before releasing sq_lock on the failure path, rewrite the still- pending SQE in place as io_uring_prep_nop with user_data = nullptr. The drain loop already skips nullptr user_data (it's the wake-up sentinel used by UringIoHandler::Wake), so when a later submit flushes this nop the CQE is drained harmlessly. Safe to mutate the SQE in place because we still hold sq_lock and no kernel/concurrent submitter has observed it yet. ### NativeDevice sector_size always returned 512 FileSystemSegmentedFile::alignment() returned a hard-coded 512. NativeDeviceImpl::sector_size() delegated to it, so the C# wrapper's sector-size cross-check in EnsureNativeDeviceCreated would: - falsely throw on 4K-native disks where ProbeAlignment returns 4096 (managed 4096 vs native 512 → 'sector-size mismatch' → device unusable), or - on 4K-native disks where the managed probe fell back to 512 (e.g. older kernel without STATX_DIOALIGN), let the device initialize with SectorSize=512 and then have the kernel reject the 512-aligned O_DIRECT buffers with EINVAL. Fix: factor the STATX_DIOALIGN probe from NativeDevice_ProbeAlignment into a shared inline helper (native_device::ProbeDioAlignment in native_device.h) and call it once from the NativeDeviceImpl ctor, caching the result as the immutable member device_alignment_. sector_size() now returns the cached value; NativeDevice_ProbeAlignment delegates to the same helper. Both sides of the ABI cross-check go through identical probe logic, so the check is now a meaningful ABI / runtime-drift detector instead of a 4K-disk footgun. ### Stale IDevice.Initialize XML The omitSegmentIdFromFilename param said it was 'only supported by managed devices — NativeStorageDevice rejects this flag'. Native devices have honored the flag since 6584cf7. Updated the doc. ### Alpine install hint The 'IoBackend.Uring with libaio fallback' error message suggested 'sudo apk add liburing' on Alpine, but README.md notes that the prebuilt won't load on Alpine (musl) at all. Replaced the apk suggestion with the actual Alpine support story (use a glibc image or fall back to a managed device). Verification: 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass on Linux. Both .so binaries rebuilt (uring-enabled and libaio-only fallback) with correct ldd output. Device.benchmark NVMe saturation throughput unchanged within noise (libaio 738K IOPS, uring 349K IOPS on Dell P5600). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: probe sector size via sysfs max(logical, physical) Replaces the statx(STATX_DIOALIGN) probe in ProbeDioAlignment with a direct sysfs lookup of max(logical_block_size, physical_block_size). Why: - STATX_DIOALIGN reports only the kernel-enforced minimum (= logical block size). It misses the firmware's preferred sector (physical_block_size), so on a 512e drive (logical=512, physical=4096) the probe would return 512 and Tsavorite would take a firmware RMW penalty on every partial-sector write. - STATX_DIOALIGN also requires kernel 6.1+ AND the filesystem to populate the field; ext4 on 6.8 leaves it unset on 512-byte devices, so the probe was already falling through to the 512 default in practice. - sysfs gives us both values directly, on every kernel, with no O_DIRECT dance. Taking max(logical, physical) covers the correctness floor (logical = kernel-enforced minimum) and the performance floor (physical = avoid RMW on partial writes) in one shot. Implementation: - stat() the file (or its closest existing ancestor — log file may not exist yet at construction). Extract st_dev → (major, minor). - Read /sys/dev/block/<maj>:<min>/queue/{logical,physical}_block_size. For partitions (e.g. sda2), the queue/ dir lives on the parent whole-disk block device — fall through to ../queue/<field>. - Round result up to a power of two (always already pow2 on real hardware) and floor at 512 B. On this machine (Dell P5600 NVMe + PERC sda): Probe(/DATA2/badrishc) = 512 (NVMe: logical=512, physical=512) Probe(/tmp/devbench) = 512 (sda partition via parent-walk) Probe(/home/badrishc) = 512 Probe(/) = 512 All values match max(logical, physical) read directly from sysfs. Verification: - 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass - Both .so flavors rebuilt (uring-enabled + libaio-only fallback) - C ABI NativeDevice_ProbeAlignment delegates to the same helper, so managed SectorSize and native sector_size() remain in lockstep. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native tests: align test buffers to 4096, matching sysfs-probed sector size CI failure on IDevice_Initialize_OmitSegmentIdFromFilename_BareFileName (and likely other IDevice_* tests on the same runner) reported: 'NativeStorageDevice.WriteAsync: misaligned I/O — sector size is 4096, but offset=0x0, length=4096, buffer=0x...7EC7A9F76800' The buffer ends at 0x...800 = 2048 — i.e. 2048-aligned but not 4096-aligned. The test helper allocated buffers aligned to HardeningSectorSize = 512 (the pre-PR default for every Garnet Linux device); a 512-aligned formula can land on a 2048-boundary that is not also a 4096-boundary. CI's underlying disk reports physical_block_size = 4096 in sysfs, so the new max(logical, physical) probe returns 4096 there. The native shim then correctly rejects sub-4096-aligned O_DIRECT buffers with EINVAL. The fix is on the test side: bump HardeningSectorSize from 512 to 4096 so the test buffer alignment matches the strictest device.SectorSize seen on any modern hardware (512n, 512e, 4Kn). Locally (Dell P5600 NVMe, logical=physical=512 → SectorSize=512) all 62 IDevice + NativeStorageDevice tests still pass — 4096 trivially divides 512. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite: revert GetAndPopulateReadBuffer changes (defer to separate PR) The read-window sizing optimization (drop leading-slop padding + clamp to page-end) is out of scope for this device-backend PR; it interacts with the larger read-IO path and deserves its own focused PR with dedicated benchmarking. Reverting to the pre-PR behavior here. TryAllocateRetryNow's bounded-backoff change is retained — it's a self-contained allocator hot-path fix and is independently verified (+13.9% on YCSB load with libaio at 64 threads, per kvbench benchmarking). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: Windows probe uses IOCTL_STORAGE_QUERY_PROPERTY for max(logical, physical) Symmetry with the Linux sysfs probe — Windows now reads both BytesPerLogicalSector and BytesPerPhysicalSector from the volume's STORAGE_ACCESS_ALIGNMENT_DESCRIPTOR (via IOCTL_STORAGE_QUERY_PROPERTY + StorageAccessAlignmentProperty) and returns the rounded-up-to-pow2 max, floor 512 B. Previously the Windows branch returned 512 unconditionally, which would silently undersize SectorSize on Windows 4Kn / 512e drives. Implementation: - Parse drive letter from filename ("C:\foo.dat" -> "\\.\\C:"). UNC paths are not supported by this probe — fall back to 512. - CreateFile on the volume with FILE_READ_ATTRIBUTES (no admin needed). - DeviceIoControl(IOCTL_STORAGE_QUERY_PROPERTY) with StorageAccessAlignmentProperty. - max(logical, physical), round up to pow2, floor 512. Linux behavior unchanged. Both .so flavors rebuilt and pass 62/62 device tests on this machine (logical=physical=512 NVMe). REQUIRES Windows DLL rebuild — the Windows path in ProbeDioAlignment is now non-trivial, and the existing prebuilt native_device.dll still returns 512 unconditionally. Without the rebuild, on a Windows 4Kn box the managed SectorSize cross-check would (incorrectly) pass at 512 while the device might actually need 4096. Rebuild recipe in the companion review comment / Tsavorite/cc/README.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add binaries * Tsavorite Native: skip wake-up/failed-submit sentinel CQEs in TryCompleteFor Addresses Copilot review comment on file_linux.cc:373. UringIoHandler::TryCompleteFor (and via it TryComplete) dispatched every drained CQE through DispatchUringCqe without checking the user_data = nullptr sentinel that QueueRunFor already handles. The sentinel marks two kinds of no-op CQEs: - Wake-up nops submitted by UringIoHandler::Wake to unblock the drainer on Dispose. - SQEs rewritten in-place after io_uring_submit failed (the SQE leak fix in c6d68925); these are committed to the SQ but carry no caller context. If a TryComplete() / TryCompleteFor() call picks up either kind of nop CQE, DispatchUringCqe would dereference the null context at context->callback(...) and segfault. Fix: mirror the nullptr-skip from QueueRunFor in TryCompleteFor. Return true to count the drain (matching the any-flag semantics). Verification: 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass. uring .so rebuilt; libaio-only .so is byte-identical because the patched code path is wrapped in #ifdef FASTER_URING and not compiled into the libaio-only fallback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native (uring): batch-drain CQEs and dispatch outside cq_lock QueueRunFor used to acquire cq_lock per CQE (peek -> read fields -> cqe_seen -> release -> dispatch). With a single drainer thread that serializes lock acquire/release on every completion and forces submitters to wait through callback latency when they need ring access. Replaced with the canonical liburing batch-drain idiom: - acquire cq_lock once - io_uring_peek_batch_cqe(ring, cqes, 64) to pull up to 64 CQEs - snapshot (io_res, context) for each - io_uring_cq_advance(ring, n) to release the slots - release cq_lock - dispatch callbacks outside the lock This is the io_uring equivalent of libaio's io_getevents(n) per syscall. Snapshot BEFORE cq_advance is mandatory because the kernel may reuse CQ slots once advanced, leaving the cqe pointers dangling. The wake-up / failed-submit sentinel (user_data == nullptr) is still skipped without dispatch, same as before. Measured impact on Dell P5600 (16 submitter threads, batch 64, throttle 256): ct=1 (1 ring, 1 drainer): 339K -> 354K ops/sec (+4%) ct=4 (4 rings, 4 drainers): 735K -> 737K (saturates, noise) ct=8 (8 rings, 8 drainers): 750K -> 742K avg (saturates, noise) The single-drainer gain is modest because the real bottleneck at ct=1 with 16 submitters is sq_lock contention on the single ring, not cq_lock contention. The batch-drain is still strictly better: - dispatches outside the lock so submitters aren't blocked by user-callback latency, - matches the idiomatic liburing pattern, - amortizes the lock acquire/release across up to 64 CQEs per cycle. For high-throughput workloads, sharding across multiple rings remains the right scaling lever (ct >= 4 saturates this drive). Verification: 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass. uring .so rebuilt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native (uring): per-thread ring affinity + 4 default rings Eliminates the sq_lock contention that was capping uring at ~340K IOPS at the default numCompletionThreads=1. Two changes work together: 1. Per-thread ring affinity in pick_ring (file_linux.h): Each submitter thread is assigned a ring on its first submit (round- robin against other threads via an atomic counter) and keeps that assignment for life. Same-thread submits never contend on sq_lock with themselves; different threads only contend when they got assigned the same ring (num_submitter_threads > num_rings). This is the user-space equivalent of libaio's "io_submit is thread-safe per io_context" — eliminate shared mutable state across submitters. 2. Hardcoded 4 rings for uring (NativeStorageDevice.cs): numIoContextsConfig = ioBackend == Uring ? max(kDefaultUringRings=4, numCompletionThreads) : numCompletionThreads So uring always has at least 4 rings even at numCompletionThreads=1. The single drainer covers all 4 rings via the legacy QueueRun compat scanner (CompletionWorker passes ctxIdx=-1 in that case). libaio is unchanged: rings == numCompletionThreads (extra rings don't help; the kernel io_context mutex is already efficient). Result on Dell P5600 NVMe (16 submitter threads, batch 64, throttle 256): Before (1 ring, 1 drainer): ~340K After (4 rings, 1 drainer, default): ~700K (matches libaio ct=1) After (8 rings, 8 drainers, sharded): ~745K (unchanged, was already saturating) No new public configuration parameters. numCompletionThreads still controls drainer count; the ring count is now backend-derived behind the scenes. The CompletionWorker single-drainer-multi-ring path was added specifically so the default numCompletionThreads=1 case can saturate without spawning extra drainer threads. Also: bumped HardeningSectorSize and the legacy bufferPool / NativeDeviceTest2 sector_size constants from 512 to 4096 to match the strictest device SectorSize we expect on any modern hardware (4Kn drives where the new max(logical, physical) probe returns 4096). Tests would otherwise fail with EINVAL on 4Kn CI runners with 512-aligned buffers. Verification: - 64/64 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass - Default uring (no flags) hits 626-740K across t=1..64 vs ~340K before - Sharded ct=4/8 unchanged (still saturates) - libaio default unchanged Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add binaries * Tsavorite Native tests: fix ReadInto length mismatch surfaced by 4096 SectorSize NativeDeviceTest1 read 1024 bytes (entryLength) using ReadInto, which: - rounded the read length up to the device sector size, - then returned a buffer of that ROUNDED length, - which the caller compared via SequenceEqual against the original `entry` byte[] (length 1024). When SectorSize was 512 (the old constant probe), 1024 rounded to 1024 and the lengths happened to match. With the new max(logical, physical) probe returning 4096 on 4Kn drives (Windows/Ubuntu CI runners), 1024 rounds to 4096, the returned buffer is 4096 bytes long, and SequenceEqual fails on length mismatch (regardless of content). Pre-existing latent bug — the rounding to sector size is correct for the IO submit, but the caller should only see the bytes it asked for. Fix: return a buffer of the caller-requested logical `size`, not the sector-rounded `numBytesToRead`. Verification: 64/64 Tsavorite.test.hlog NativeDeviceTest + IDevice + NativeStorageDevice tests pass on Linux (where SectorSize is 4096 on the CI runner's 4Kn drive). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite benchmarks: refresh stale completion-threads help text The --device-completion-threads (KV.benchmark) and --completion-threads (Device.benchmark) help text said "all drainers share the same kernel io_context / io_uring" and "values > 1 are rarely useful past 1 today". Both claims are stale since the sharded-rings work (8cbca9d4d) and the per-thread ring affinity + 4-default-rings change (298bfd180): - Each drainer is bound 1:1 to its own kernel io_context (libaio) or io_uring ring (uring). - Submitters distribute across rings via per-thread affinity. - For io_uring, throughput scales with completion-threads up to available submitter concurrency (measured: ct=1 ~340K → ct=4 ~735K on Dell P5600 NVMe at the device-benchmark level). - For libaio extra drainers still rarely help past 1 (kernel per-context mutex is efficient). - Note added that uring uses min 4 rings even at ct=1 with the single drainer covering all rings via the legacy QueueRun scanner. Help-text-only change. No code behavior change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: fix KV.benchmark deadlock on multi-segment disk-spill reads Root cause: cross-segment read rejection + engine retry loop ============================================================== The AllocatorBase.GetAndPopulateReadBuffer sector-aligned read window can extend past the page-end boundary when reading a record near the tail of a page. When the device's segment size is a multiple of the page size (e.g. 4MB pages, 1GB segments — the Garnet default), an over-extended read at the last page of a segment also crosses the device's segment boundary. NativeStorageDevice's underlying FileSystemSegmentedFile rejects cross-segment reads with Status::IOError; the engine's AsyncGetFromDiskCallback interprets a 0-byte read as a short read and retries the same address — forever. Worker thread spins at 99% CPU, disk activity drops to zero, benchmark deadlocks. Reproduced reliably on KV.benchmark: --device native --device-io-backend libaio \ --log-memory 16m --page-size 4m --segment-size 1g \ -n 10000000 (1.28 GB dataset → crosses 1GB segment boundary) Smaller datasets (1M = 128MB, fits in 1 segment) work; larger ones hang. RandomAccess device works on all dataset sizes because its managed segmented-file wrapper doesn't reject cross-segment reads. Diagnostic captured the exact symptom: a read at sourceAddress 0x3FFFF600 (1,073,739,776 — 2,560 bytes before the 1GB segment boundary) with readLength 4608 (sector-aligned record window) extends to 0x40000C00 — 2,560 bytes into segment 1. Native rejects with Status::IOError, callback fires with numBytes=0, engine retries. Fix: clamp the aligned read length so it never crosses page-end. ============================================================ Added in AllocatorBase.GetAndPopulateReadBuffer: var pageEndInFile = (ulong)(AlignedPageSizeBytes * (GetPage(fromLogicalAddress) + 1)); if (alignedFileOffset + alignedReadLength > pageEndInFile) alignedReadLength = (uint)(pageEndInFile - alignedFileOffset); Records never span page boundaries (HandlePageOverflow guarantees), so the actual record is fully readable within the clamped window — available_bytes reflects what we actually got from disk, and the engine continues normally. pageEnd is sector-aligned (PageSizeBits >= sector size), so the clamped length stays sector-aligned. Also reverted the uring "min 4 rings even at ct=1" experiment ============================================================= The earlier "default 4 rings for uring regardless of ct" change was fundamentally broken: with per-thread submit affinity (pick_ring's thread_local index), submitters bound to rings 1-3 never get their completions drained because the single drainer blocks on ring 0 with a 1-second QueueRun timeout and only briefly polls the other rings between wake-ups. The result is ~50x throughput degradation on workloads where submitters land on rings != 0 (KV.benchmark load phase dropped from 2.5M ops/sec to 54K ops/sec at t=1). Reverted to the simple rule: rings == numCompletionThreads. For uring perf scaling, users set numCompletionThreads >= expected submitter concurrency; each ring is then continuously drained by its dedicated drainer thread. Defense-in-depth hardening ========================== - NativeStorageDevice._callback now catches ALL exceptions from the user callback (was: try/finally but exception propagated). A managed exception escaping back into native code across the C ABI boundary silently terminates the drainer thread; the next submitter then spins forever in device.Throttle(). Now the exception is logged and swallowed so the drainer survives. - NativeStorageDevice.CompletionWorker has the same try/catch around the whole drain loop as defense-in-depth against unrelated managed exceptions (P/Invoke marshalling, IntPtr.Zero races with Dispose, etc.). - file_linux.cc QueueFile::ScheduleOperation (libaio) and UringFile::ScheduleOperation (uring) now retry submit-side EAGAIN indefinitely with bounded backoff (64 sched_yields, then 1ms nanosleeps) instead of returning Status::IOError after 8 yields. Surfacing transient EAGAIN as a permanent error creates the same retry-loop pathology as the cross-segment-read bug above. EAGAIN is the kernel saying "ring is full, try later"; it's not a real error and must not be exposed to the engine. Verification ============ KV.benchmark, 100M keys × 100B, 16MB log (mostly disk-spill), 1 completion thread, 100% reads: libaio: t=1 135K ops/sec, t=4 400K, t=8 444K, t=16 445K, t=32 404K uring: t=1 124K ops/sec, t=4 244K, t=8 265K, t=16 278K, t=32 272K Both backends stable across the full thread × dataset sweep (previously native+libaio hung on any 10M+ dataset; native+uring hung on every config). 64/64 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> | 3 个月前 | |
Update native storage device on Azure Linux docker container (#1924) * Fix native storage device on Azure Linux (libaio.so.1t64 SONAME mismatch) The committed libnative_device.so is built on Ubuntu 24.04, so its libaio DT_NEEDED is "libaio.so.1t64" (the 64-bit time_t ABI rename). Azure Linux, RHEL, and Fedora ship only "libaio.so.1", so the library failed to load and the native storage device was broken on those distros. This regressed in #1831, which rebuilt the binary on a t64 host; before that it needed "libaio.so.1", which those distros provide. Fix, two layers: - Dockerfile.azurelinux: add a "libaio.so.1t64 -> libaio.so.1" compat symlink and run ldconfig so the native library resolves at the system level (mirrors how the Ubuntu Dockerfiles bridge the naming). - NativeStorageDevice.cs: make the C# loader's compat-symlink fallback bidirectional. It now detects which SONAME the loader could not resolve ("libaio.so.1t64" or "libaio.so.1") and drops a symlink of that name next to libnative_device.so, pointing at whichever libaio the host actually provides (searches both t64 and plain multiarch/lib64 paths). Previously it only handled the reverse direction, which became backwards after #1831. This also covers non-Docker RHEL/Fedora/Azure Linux users. Add a musl guard (IsMuslRuntime, detected via /lib/ld-musl-*.so): on Alpine the glibc-built .so cannot load, so skip the shim instead of forcing it to bind against musl libaio and segfault, restoring the clean managed-device fallback. Validated with test/docker-tests/validate_docker_images.py across all five Linux images: 70 passed, 0 failed. No IO hot-path or native binary changes, so device throughput/latency are unchanged; on Azure Linux the native device now loads instead of failing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c54dbeb2-83d6-499b-a45f-a39368148b80 * Address review: make libaio diagnostic path distro-accurate The BuildLibaioDiagnostic "ln -sf" hint derived the library directory from the Debian/Ubuntu multiarch triplet (/usr/lib/<triplet>) whenever the CPU arch was known. But this diagnostic fires precisely on non-t64 glibc distros (RHEL/Fedora/Azure Linux), where libaio lives under /usr/lib64 or /usr/lib, so the suggested path was misleading on exactly those hosts. Derive the hint from the libaio the host actually ships instead: extract the candidate search into TryFindHostLibaio() (reused by the shim), and build the example command from the real path (e.g. /usr/lib64/libaio.so.1 on RHEL, or /usr/lib/libaio.so.1 on Azure Linux). When no libaio is present, point at the install step rather than a bogus symlink path. Removes the now-unused TryGetLinuxMultiarchTriplet helper. Diagnostic-text/refactor only; device load behavior is unchanged. Validated: azurelinux + alpine docker images 28 passed, 0 failed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c54dbeb2-83d6-499b-a45f-a39368148b80 * Address review: tighten IsUsableLibaioShim SONAME match The shim-usability check used target.Contains("libaio.so.1"), which is looser than intended: it would also match an unrelated future SONAME such as "libaio.so.10". Compare on the target's file name and accept only the two supported SONAMEs ("libaio.so.1t64", "libaio.so.1") or a versioned "libaio.so.1.*" real file, which rejects "libaio.so.10"/"11"/"100" while still handling relative or absolute link targets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c54dbeb2-83d6-499b-a45f-a39368148b80 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> | 1 个月前 | |
[Tsavorite] Add native Linux storage backend, harden NativeStorageDevice, refresh storage benchmarks (#1831) * Port device/IO changes from optimize-v2-io onto kv-bench All 31 non-benchmark files changed on optimize-v2-io (vs its branch base d3677cfaa) ported here. Backup tag: optimize-v2-io-prerebase-backup @ 3f41f2bdf. Scope: device/IO/native-backend ONLY. Includes: Tsavorite C++ native device: - io_uring backend + pluggable C ABI (file_linux.cc/h) - error model split (native_device_error.h) - file_system_disk + native_device.h updates - CMakeLists + README Tsavorite C# device: - NativeStorageDevice: IoBackend enum (Default, Libaio, Uring), completion threads, production-readiness pass - LinuxFileExtensions.cs: P/Invoke open() for true O_DIRECT - ManagedLocalStorageDevice + RandomAccessLocalStorageDevice: O_DIRECT wiring on Linux - Devices.cs: router updates for new device APIs Tsavorite allocator + utilities: - AllocatorBase: bounded backoff in TryAllocateRetryNow - CompletionEvent: Wait(TimeSpan) overload Tsavorite checkpoint management: - LocalStorageNamedDeviceFactory + Creator surface ioBackend + completionThreads parameters Tests: - DeviceTests.cs updated for new device APIs Garnet host: - --device-io-backend, --device-completion-threads flags - defaults.conf updated, GarnetServerOptions wiring Dockerfiles (all 5): install liburing alongside libaio. Note: YCSB.benchmark and KV.benchmark are not modified by this commit; KV.benchmark is the supported benchmark on this branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * KV.benchmark: add uring backend, fix --validate for disk-spill, doc liburing runtime dep - Options: --device-io-backend now accepts 'uring' (aliases: io_uring, iouring) in addition to libaio/default; help text + validation error list updated to match. - KV.benchmark README: device-backend table now describes the libaio vs uring split, with a runtime-install snippet for liburing across Debian/Ubuntu, Fedora/RHEL/AzureLinux, and Alpine, plus link to the full Tsavorite Native Device docs. New 'native + libaio' and 'native + uring' rows added to the cookbook for constrained-log large-dataset workloads. - Tsavorite Native Device README: new top-level 'Runtime dependencies (end users)' section listing the apt/dnf/apk install lines and how to fall back to the no-liburing variant. - KV.benchmark Validate: fix two bugs that surface when load and run use different thread counts and when the log spills to disk: 1) writerThread reconstruction now uses ResolvedLoadThreads (not Options.Threads which is the RUN count), so --load-threads N with --threads M != N validates correctly. 2) Reads of records below HeadAddress return Status.IsPending; the previous code counted these as misses. Validate now issues reads in batches of 256 and drains via CompletePendingWithOutputs, verifying each completed output against the per-thread pattern. Verified end-to-end with both backends: 4.6M × 100B × 8T × log=256m (~580MB dataset > 256MB log → forces disk spill) × 50R/50U × --validate: native + libaio → [validate] OK, run = 1.27 M ops/s native + uring → [validate] OK, run = 1.02 M ops/s 4.6M × 100B × 8T × log auto (fits) × 95R/5U × --validate: native + libaio → [validate] OK, run = 16.10 M ops/s native + uring → [validate] OK, run = 16.33 M ops/s Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite: replace MarkHandleAsAsync reflection hack with RandomAccess on SafeFileHandle Background: MarkHandleAsAsync used reflection to flip SafeFileHandle.IsAsync's non-public setter so a P/Invoke-opened O_DIRECT FD could be wrapped in 'new FileStream(handle, isAsync: true)' without throwing 'Handle does not support asynchronous operations'. The flag is non-public in .NET 8/10, the hack was fragile across future runtime versions, and on Linux IsAsync is a contract gate (no real overlapped I/O exists for files), so the lie bought us nothing beyond letting the FileStream constructor accept the handle. RandomAccessLocalStorageDevice — refactor: - StorageAccessContext.handle is now SafeFileHandle (was FileStream). - CreateRead/WriteHandle: * Linux + O_DIRECT capable: LinuxFileExtensions.OpenDirect -> raw SafeFileHandle, no FileStream wrap. Page-cache bypass via the O_DIRECT flag at open(2), exactly as before. * Otherwise (Windows; or Linux when filesystem rejects O_DIRECT): File.OpenHandle(path, ..., FileOptions.Asynchronous | cast FILE_FLAG_NO_BUFFERING). On Windows this gives the runtime IOCP-bound OVERLAPPED I/O; on Linux it's page-cached. - All I/O goes through RandomAccess.{Read,Write}Async(safeHandle, memory, offset). On Windows: true kernel async via IOCP. On Linux: pread/pwrite dispatched to ThreadPool (same as before). - GetFileSize uses RandomAccess.GetLength(handle). - SetFileSize uses RandomAccess.SetLength(handle, size). LinuxFileExtensions: - MarkHandleAsAsync and the IsAsyncProperty reflection are deleted entirely (no remaining callers). - System.Reflection using removed. ManagedLocalStorageDevice: - Reverted to origin/main. This device is designed to stay within FileStream APIs; the O_DIRECT branch we added doesn't belong here. Verified: - Tsavorite test.hlog DeviceTests: 36/36 passed. - KV.benchmark --device randomaccess --log-memory 256m --preallocate-log --rumd 50,50,0,0 --validate: PASS, 357 K ops/sec, iostat shows 100-177 K real disk r/s and 53-90% NVMe util → O_DIRECT page-cache bypass confirmed. - KV.benchmark --device randomaccess (log fits) --validate: PASS, 15.7 M ops/sec. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite tests: cross-device hardening suite (Native + RandomAccess + ManagedLocal) The Phase-7 hardening suite added in this branch was NativeStorageDevice- specific. Add parametrized variants of the four tests that are pure IDevice contract checks (not native-specific lifecycle/API), so they exercise all three local-storage device implementations: - Hardening_AllDevices_RoundTrip_BasicReadWrite - Hardening_AllDevices_RoundTrip_AcrossSegmentBoundary - Hardening_AllDevices_Parallel_32ConcurrentWrites - Hardening_AllDevices_Parallel_BurstyTraffic Each is parametrized by a new DeviceKind enum (Native, RandomAccess, ManagedLocal). Native is gated on OperatingSystem.IsLinux() (the C++ shim links against libaio/liburing); the other two run on both Linux and Windows. A shared CreateDeviceForTest helper takes care of the per-kind ctor + Initialize() dance so the test body stays uniform. Result: 38/38 hardening tests pass on Linux (12 new cross-device + 26 native-only). Native-specific tests retained as-is because they test API that doesn't exist on the other devices: - Lifecycle (DisposeBeforeInitialize, InitializeTwice, etc.) — NativeStorageDevice defers Initialize from the ctor; the other devices initialize in their ctor. - Segment-size validation (NonPowerOfTwoSegmentSize_Throws, etc.) — Initialize() is the only callsite that validates. - Recovery_*_SegmentSize_* — the native device's open() path is the only place that records and re-validates per-segment-size metadata. - SectorSize stability across opens — not all devices expose this. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite device tests: split into IDevice_ contract + NativeStorageDevice_ buckets; fix AsyncPool creator-throw hang Test refactor: - IDevice_*: 8 contract tests parametrized across Native, RandomAccess, ManagedLocal (round-trip basic, round-trip cross-segment, round-trip various segment sizes, 32 concurrent writes, 64 concurrent reads, mixed reads+writes, bursty traffic, stress burst of 100 writes, permission-denied callback contract). 33 cases total. - NativeStorageDevice_*: 16 native-only tests for behaviors managed devices don't have (deferred Initialize signature, recovery segment-size mismatch detection, sector-size discovery, sync-throw unaligned IO guard). AsyncPool fix: GetOrAdd reserved a slot in totalAllocated before calling creator(). If creator() threw (e.g. open() returned EACCES, ENOSPC), the slot was never released, so Dispose() would loop forever waiting for totalAllocated to drain to zero. This manifested as a process hang when a device pool's first open() failed. Rollback the reservation on exception so the failure propagates cleanly and the pool can still be disposed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite IDevice: unify Initialize contract — required for all devices, -1 = unbounded single segment Before this change, IDevice.Initialize had the same signature for every device but very different semantics: * StorageDeviceBase (RandomAccess, ManagedLocal, LocalStorage, NullDevice, LocalMemoryDevice): the ctor pre-set segmentSize = -1 / bits = 64 / mask = ~0, so calling an IO entry point without Initialize() silently ran in unbounded single-segment mode. * NativeStorageDevice: Initialize was MANDATORY (the C++ shim needs the segment size at create time for libaio/io_uring geometry), IO entry points threw if invoked first, and segmentSize = -1 was rejected. This commit unifies the contract: every IDevice must call Initialize() exactly once before any IO, and segmentSize = -1 selects unbounded single-segment mode on every device. Implementation: * StorageDeviceBase - Added `initialized` flag (volatile) and `EnsureInitialized()` helper that throws InvalidOperationException with a clear message naming the device by FileName. - Ctor leaves `initialized = false` but keeps the safe fallback defaults (-1 / 64 / ~0) so any cold maintenance path that touches segmentSizeBits before the guard can't compute outright nonsense. - EnsureInitialized() called from the base address-based ReadAsync / WriteAsync overloads and TruncateUntilAddress / TruncateUntilAddressAsync. - Initialize sets `initialized = true` at the end. * NativeStorageDevice - Accepts segmentSize = -1: translates to 1UL << 63 for the native shim so the C++ FileSystemSegmentedFile's shift = log2(segment_size) math collapses every non-negative upper-layer address into segment 0 (parity with the managed-side bits = 64 / mask = ~0). Single growing file on disk. - Tracks the value passed to native in nativeSegmentSizeBytes (replaces the diagnostic-only configuredSegmentSizeBytes long field, which couldn't hold 1<<63 without overflow). - ABI readback (NativeDevice_GetSegmentSize) compared against the value we sent to native, not the user-facing -1. - Always rejects omitSegmentIdFromFilename — the C++ shim has no omit-suffix code path, every segment is written as <base>.<segmentId>. Better to fail fast than silently produce wrong file names. * Concrete IO entry points (ReadAsync / WriteAsync / RemoveSegment / RemoveSegmentAsync) of NullDevice, LocalMemoryDevice, ManagedLocalStorageDevice, RandomAccessLocalStorageDevice, LocalStorageDevice, AzureStorageDevice, ShardedStorageDevice, and TieredStorageDevice now call EnsureInitialized() before doing work. Caller fix-ups: * LocalStorageNamedDeviceFactory.Get now calls device.Initialize(-1L) before returning. Commit / checkpoint metadata is single growing-file usage (segment 0 only, .0 suffix), so unbounded mode is the right default and unblocks every DeviceLogCommitCheckpointManager caller from needing to remember to initialize. * LocalStorageNamedDeviceFactory.ListContents skips dotfile entries — defensive against a pre-existing race in LinuxFileExtensions.IsDirectIOSupported where a .tsavorite-odirect-probe-* temp file can leak in the commit dir if File.Delete races with File.GetFiles. Without this filter, leaked probe files surface as Int64.Parse("") failures in DefaultCheckpointNamingScheme.CommitNumber. * SimulatedFlakyDevice.Initialize now propagates to the wrapped device. * ComponentRecoveryTests Setup_* helpers call Initialize(-1) on devices they construct directly (bypass the Tsavorite allocator path which normally Initializes). Tests (DeviceTests.cs): * IDevice_ReadAsyncBeforeInitialize_Throws(kind) × 3 — new contract test. * IDevice_WriteAsyncBeforeInitialize_Throws(kind) × 3 — same. * IDevice_Initialize_SegmentSizeMinusOne_UnboundedSingleSegment(kind) × 3 — write at offset 1 MiB (would be in segment-N for any positive size) and read back, confirming -1 routes through segment 0 on all 3 kinds. * NativeStorageDevice_Initialize_OmitSegmentIdFromFilename_Throws — new native-only test for the omit rejection in both -1 and explicit-size modes. * Removed NativeStorageDevice_{Read,Write}AsyncBeforeInitialize_Throws (now subsumed by the IDevice_ variants). Docs: * IDevice.Initialize docstring rewritten to spell out the new contract and the -1 semantics. NativeStorageDevice.Initialize remarks updated. Verified on Linux net10.0 Release: * 599 hlog tests (491 passed + 108 skipped) * 305 recovery tests * 144 + 155 + 127 + 346 = 772 other Tsavorite + Garnet RespTests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite: native devices honor omitSegmentIdFromFilename; O_TMPFILE probe (no race) Two related fixes on top of the unified Initialize contract: 1) NativeStorageDevice now supports omitSegmentIdFromFilename ───────────────────────────────────────────────────────── Previously Native rejected the omit flag because the C++ shim hard-coded the '.<segmentId>' suffix in three places in file_system_disk.h. This made the IDevice contract asymmetric (managed devices honored omit, Native didn't). Fix by threading the bool through the entire C++/C ABI: C++ (libs/storage/Tsavorite/cc/src/device/): * FileSystemSegmentBundle: new bool omit_segment_id_; both ctors accept it and use a new segment_path(idx) helper that returns just filename_ when set, otherwise filename_ + '.' + std::to_string(idx). Used at all three locations that previously hard-coded the suffix. * FileSystemSegmentedFile: new bool omit_segment_id_ (const) wired through ctor and propagated to bundles allocated by OpenSegment. * NativeDeviceImpl: new bool omit_segment_id constructor param; recorded as omit_segment_id_ member. ValidateRecoveredSegments short-circuits in omit mode (single bare-named file, segment-size mismatch check is meaningless when there's no .<id> suffix to scan for). * native_device_wrapper.cc / NativeDevice_CreateWithBackend: new trailing 'bool omit_segment_id' parameter. ABI BUMP — managed wrapper updated to match; Linux .so rebuilt and committed at libs/storage/Tsavorite/cs/src/core/Device/runtimes/linux-x64/native/ libnative_device.so. **Windows DLL must be rebuilt by user** with cmake -G 'Visual Studio 17 2022' -A x64 -T v143,spectre=true. C# (libs/storage/Tsavorite/cs/src/core/Device/): * NativeStorageDevice P/Invoke signature updated. * NativeStorageDevice.Initialize removes the 'always rejects omit' guard. It now accepts omit:true together with segmentSize = -1 and forwards to native; rejects omit:true together with a positive segmentSize with a clear error message (multiple segments would collapse onto the same on-disk path and clobber each other). Tests (DeviceTests.cs): * IDevice_Initialize_OmitSegmentIdFromFilename_BareFileName(kind) × 3: writes via Initialize(-1, omit:true) and asserts the on-disk file is the bare basename (no .0 suffix). Replaces the native-only 'throws' test from the previous commit. * IDevice_Initialize_OmitSegmentIdFromFilename_WithoutMinusOne_Throws(kind) × 3: enforces the no-positive-size-with-omit invariant on every kind. 2) IsDirectIOSupported uses O_TMPFILE (race-free probe) ───────────────────────────────────────────────────── The previous probe in libs/storage/Tsavorite/cs/src/core/Device/ LinuxFileExtensions.cs created a hidden '.tsavorite-odirect-probe-<pid>- <guid>' file in the device's directory, then File.Delete'd it in a silent-catch finally. Multiple concurrent commits (one device per Get()) ran probes simultaneously; concurrent ListContents calls from CommitRecordBoundedGrowthTest would observe the probe file during its brief lifetime, and DefaultCheckpointNamingScheme.CommitNumber would then throw FormatException on long.Parse(''). 18/20 baseline failure rate. Switching from create+unlink to open(directory, O_TMPFILE | O_RDWR | O_DIRECT) tells the kernel to allocate an anonymous inode in the directory's filesystem with NO directory entry. The probe inode is invisible to readdir/getdents regardless of timing; concurrent ListContents cannot observe it. Freed on close. Linux >= 3.11 + ext4/xfs/tmpfs/btrfs all support it. If O_TMPFILE itself fails (EOPNOTSUPP on some filesystem) we conservatively report 'no O_DIRECT' so the device falls back to the page-cache path — no named-file fallback because that's the bug we're fixing. Reverts the dotfile filter in LocalStorageNamedDeviceFactory.ListContents added by the previous commit; the underlying race is now eliminated at the kernel level so the workaround is unnecessary. 20/20 LogFastCommitTests runs pass after the change (was 2/20 on baseline, 3/20 on the previous unlink-after-open attempt which still raced). Verified on Linux net10.0 Release: * 612 hlog tests (504 passed + 108 skipped) * 305 recovery tests * 62 device tests (IDevice contract + NativeStorageDevice-specific) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * KV.benchmark: add disk-IO thread-scale sweep recipe to cookbook Captures the 100M-key disk-bound thread-scale experiment from the optimize-device branch sweep so it can be reproduced verbatim in the future: - 100M × 100B records (12.8 GB on disk) - 16 MB log so ~0.125 % of dataset is in memory and almost every read is a 4 KB random disk fetch - 8 load threads, run-threads sweep 1,2,4,8,16,32 at 15s each - One row per backend (RandomAccess / native+libaio / native+uring) Added a short note after the table explaining what to compare against (the disk's fio ceiling at 4K-aligned QD=64-per-job), the expected ~2 min wall-clock per device, and the observed per-backend plateau characteristics so the next operator knows what 'good' looks like. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Device.benchmark + KV.benchmark: robustness & flag improvements Device.benchmark fixes (previously reported throughput could be 2x inflated): - Throughput counter now tallies successful completions only. Before, every ReadAsync call was counted as success even when the kernel returned EAGAIN (Status::IOError=4 — flooded libaio io_context ring). Under --throttle-limit 0 with high QD, ~40% of "ops" were errored requests. - Per-error-code histogram printed at end of run; no per-error Console.WriteLine (was emitting millions of serial writes/run, both falsifying numbers and slowing the real path). - DEBUG data validation skipped on errored ops (was reading garbage from the destination buffer on EAGAIN paths and reporting spurious "Data mismatch"). - --throttle-limit help text documents the libaio kernel ring trap (128 slots wide; high QD + no throttle floods it) and recommends --throttle-limit 128 (also the io_uring SQ depth this build uses). - --io-backend flag added (libaio / uring / default) so the existing Linux Native path can be exercised against either backend. Unknown values are rejected with an actionable message at startup instead of silently falling back to default. - --completion-threads is now wired through to the Linux Native ctor (was hardcoded to 1). - --file-size widened to long (was int; --file-size > 2GB threw a parse error). KV.benchmark + Devices.cs: clarified XML/help text for the existing --device-completion-threads / numCompletionThreads parameter to describe the current behavior (multiple drainer threads share one kernel io_context / io_uring per device). No behavior change for KV.benchmark or core Tsavorite Devices.cs API beyond the Device.benchmark surface and clarified docs. * Tsavorite Native: shard io_uring per completion thread; new C ABI Adds N independent kernel io_contexts (libaio) / io_urings (uring) per NativeStorageDevice, with completion threads bound 1:1 to contexts. libaio internally always uses 1 context (sharding empirically gave nothing — kernel mutex efficient at all tested loads), but the sharded ABI surface is kept across both backends so the templated NativeDeviceImpl<HandlerT> doesn't need a fork. C ABI changes (libnative_device.so): - NativeDevice_CreateWithBackend signature bumped: trailing int32 num_io_contexts. - New exports: NativeDevice_QueueRunFor(device, ctx_idx, timeout_secs), NativeDevice_NumIoContexts(device). - Legacy NativeDevice_QueueRun kept; under uring sharding it scans all rings (back-compat for any single-thread drainer). C# (NativeStorageDevice): - Synchronous ABI probe at Initialize() that converts EntryPointNotFoundException into a clear TsavoriteException listing the missing exports and how to rebuild — guards against a stale .so silently hanging Dispose's drain loop. - Probe is intentionally gated by the QueueRun branch so it runs only on backends that actually use the new symbols at runtime: Linux Native (libaio / uring) where QueueRun returns >= 0, NOT on Windows IOCP where the ThreadPoolIoHandler returns -1 by design. This means a stale Windows DLL keeps working unchanged because it never calls the sharded exports. cdecl/x64 ABI silently tolerates the new trailing num_io_contexts arg on NativeDevice_CreateWithBackend. - Completion threads bound 1:1 via QueueRunFor(ctxIdx) (no closure capture bug — ctxIdx is captured per-iteration into a local). - numCompletionThreads is the user-facing knob; native side decides how many contexts to actually create (libaio: always 1; uring: honours the request). Empirical justification (Device.benchmark, NVMe, 4K random reads, batch=4096, throt=512, fio ceiling 749K, NUMA0-pinned): libaio CT=1 (always 1 ctx + 1 drainer): 755K ops/sec (t=32) uring CT=1 (1 ring + 1 drainer): 357K ops/sec ← SpinLock-bound uring CT=1 ring + N drainers (regresses): 357K → 274K ← cq_lock contention uring CT=4 (4 rings + 4 drainers): 745K ops/sec uring CT=8 (8 rings + 8 drainers): 758K ops/sec ← hardware ceiling Both backends now reach the hardware NVMe ceiling. uring requires sharding (the user-space SpinLock around io_uring_get_sqe + prep + submit is the real cap). libaio doesn't need sharding (kernel io_context mutex already efficient at all tested loads). Files: - file_linux.h : UringIoHandler sharded (vector<io_uring*>, per-ring sq_lock + cq_lock, atomic round-robin pick_ring). QueueIoHandler unchanged on the data plane (single io_context_t) but exposes the same num_contexts()/TryCompleteFor/QueueRunFor surface as inline stubs for ABI symmetry. - file_linux.cc : new UringIoHandler impls; QueueIoHandler unchanged. - file_windows.h: stub overloads (num_contexts()=1, QueueRunFor=-1, 2-arg ctor) so the templated NativeDeviceImpl compiles unchanged. - native_device.h, native_device_wrapper.cc: ABI plumbing as above. - NativeStorageDevice.cs: ABI probe + per-context drain workers. - runtimes/linux-x64/native/libnative_device.so: rebuilt with sharding. * Device.benchmark: add cookbook README showing how to saturate ~750K NVMe IOPS Captures the verified copy-paste recipe for both Linux Native backends (libaio and io_uring) to hit the hardware ceiling on a Dell P5600-class NVMe, alongside a flag reference, output-schema explanation, and troubleshooting table. Headline recipes verified end-to-end on the reference setup: libaio --completion-threads 1 --threads 16 --throttle-limit 512 → 743K ops/sec uring --completion-threads 8 --threads 16 --throttle-limit 512 → 738K ops/sec (Both within 2 % of the table values in the README; zero kernel-side errors.) Key facts documented: - libaio always uses one io_context in this build regardless of --completion-threads (sharding empirically gave nothing; the hint is ignored). Pass 1 explicitly so scripts are self-describing. - io_uring needs sharded rings (CT >= 4) to escape the per-ring user-space SpinLock cap around io_uring_get_sqe + prep + submit; CT=8 is the safe peak. - --throttle-limit must be set to at least the per-ring/per-context depth (128 in this build for both backends). --throttle-limit 0 floods the kernel ring and the benchmark correctly surfaces Status::IOError=4 in the per-code histogram rather than tallying errored ops as throughput. - --file-size must be a multiple of 1024 × --sector-size (fill phase uses a 1024-sector temp buffer). Plus a section comparing Device.benchmark vs KV.benchmark to direct readers to the right tool: Device.benchmark for IO-layer ceiling validation (saturates NVMe), KV.benchmark for full-path throughput (currently caps ~30 % below the IO ceiling on the upper-layer pending-read path — see KV.benchmark README for that side of the story). Also adds a one-paragraph pointer in benchmark/README.md so the new README is discoverable from the top-level benchmarks listing. * Tsavorite Native: bounded sched_yield retry on transient kernel-ring full ScheduleOperation in both QueueFile (libaio) and UringFile (io_uring) now retries the kernel-side submission on the transient back-pressure signal (libaio: `io_submit == 0`; uring: `io_uring_get_sqe == nullptr`) up to kMaxSubmitRetries = 8 attempts, each separated by a sched_yield(). Permanent errors (libaio io_submit < 0, uring io_uring_submit < 0) are NEVER retried — they surface immediately as Status::IOError. Motivation: the upper-layer throttle gate in AllocatorBase.AsyncGetFromDisk is a racy test-then-increment (Throttle() reads numPending non-atomically, then ReadAsync does Interlocked.Increment). With N concurrent submitters all passing the gate at numPending == ThrottleLimit, in-flight can spike to ThrottleLimit + N momentarily, exceeding the 128-slot per-context / per-ring kernel ring depth when N > 8 (which is normal for Garnet under heavy disk-bound load). Pre-fix, the kernel rejects the overshoot submissions with EAGAIN, which Tsavorite handled by re-routing through the full AllocatorBase pending-read retry loop (correct but expensive: a full round-trip per IOError). Post-fix, the burst is absorbed locally by a handful of sched_yields and never surfaces to the upper layer. Why sched_yield + bounded retries is the right shape: on a 750K-IOPS NVMe the kernel ring drains a slot every ~1.3 µs and sched_yield is typically 1-10 µs on Linux, so 8 retries (worst-case ~40-80 µs window) is more than enough to absorb the typical 24-slot overshoot from a 32-thread burst. For genuine sustained overload (application submission rate exceeds device IOPS for seconds), the retries exhaust and Status::IOError surfaces — which is the correct signal for the caller to apply back-pressure. Implementation notes: - libaio: simple loop around io_submit. No lock to release; io_submit is kernel-thread-safe per io_context, so concurrent submitters serialise inside the kernel. - uring: must release sq_lock around sched_yield. Holding a SpinLock across a syscall would stall every other submitter on the same ring. Only the get_sqe == nullptr path is retried; if get_sqe succeeded we've already "consumed" an SQE slot in the user-side bookkeeping and re-issuing via get_sqe+prep on retry would corrupt the ring (we'd hold two SQEs for one logical op). For SQPOLL-disabled rings (our setup) io_uring_submit returns 1 in steady state — a non-1 there is an unrecoverable kernel-side error and surfaces immediately. Verified end-to-end (Device.benchmark, NVMe, 4K random reads, batch=4096, NUMA0-pinned, --throttle-limit 120 to match production NativeStorageDevice default): libaio CT=1: t=8 / 16 / 32 → 626K / 631K / 627K ok/sec, 0 err uring CT=8: t=8 / 16 / 32 → 622K / 625K / 626K ok/sec, 0 err At extreme intentional-overload settings (--throttle-limit 4096, t=32) errors still appear — confirming the retry budget correctly exhausts when the application is genuinely outpacing the device: libaio CT=1, t=32, throt=4096: 754K ok, 14.7M code4 err (~5% of submits) uring CT=8, t=32, throt=4096: 745K ok, 0 err (uring still produces 0 err under same overload because 8 rings × 128 = 1024 SQ slots is large enough that even gross over-submission fits within the retry budget per ring.) * Tsavorite Native: PR review fixes (3-model code review pass) BLOCKER fixes: - NativeStorageDevice.Dispose UAF race. Previously NativeDevice_Destroy(nativeDevice) ran before nativeDevice was nulled, so a concurrent guard-bypassed P/Invoke could observe a non-zero handle that points to freed memory. Fix: Interlocked.Exchange atomically captures-and-nulls the handle; destroy runs on the captured pointer. EnsureReadyOrSilent now checks disposedFlag first. HIGH fixes: - UringFile::ScheduleOperation SQE leak on transient io_uring_submit failure. After a successful get_sqe + prep, the SQE is committed to the user-side SQ ring; a -EAGAIN/-EBUSY return from io_uring_submit left the slot permanently occupied with no kernel iocb, eventually starving get_sqe forever. Fix: retry io_uring_submit (without re-preparing) up to kMaxSubmitRetries on transient negatives, with sched_yield (and sq_lock released) between attempts. - UringIoHandler::Init partial-init leak. If new SpinLock() threw after io_uring_queue_init succeeded for ring i, the already-initialized ring leaked (the class dtor doesn't run on partial construction). Fix: use std::unique_ptr RAII holders during construction; release into the member vectors only after all allocations succeed. POLISH fixes: - NativeStorageDevice.Initialize tail-throw cleanup. If base.Initialize threw after the native device and completion threads were created, both leaked. Now wrapped in try/catch that cancels token, joins threads, and destroys the native device. - UringIoHandler rule-of-5 hygiene: explicitly deleted copy ctor, copy-assign, and move-assign so the implicit shallow copies (which would double-delete the raw owning pointers) cannot be generated. - DispatchUringCqe: added static_assert(is_trivially_destructible<IoCallbackContext>) so a future non-trivial member fails the build instead of silently leaking. - NativeDeviceImpl::num_io_contexts: removed unnecessary const_cast (the underlying num_contexts() is already const on both backends). - Removed duplicate XML <summary> on NativeStorageDevice.Dispose. - FileSystemDisk dead-code ctor: passed the now-required 5th arg to FileSystemSegmentedFile so the file compiles if anyone instantiates it. Comment hygiene sweep (per explicit review-rule #4 "comments should not refer to design thought processes"): - Removed embedded benchmark results, hardware-specific throughput numbers, and historical narrative from class/method documentation in file_linux.{h,cc}, native_device.h, native_device_wrapper.cc, NativeStorageDevice.cs. - Kept WHAT each method does and the invariants it enforces; moved WHY this approach was chosen out of source comments (the commit log is the appropriate place for that context). - Net: -162 lines across 7 files, no behavior change from the trim itself. Verified post-fix performance unchanged (Device.benchmark, NVMe, 4K random reads, batch=4096, NUMA0-pinned): libaio CT=1 t=16 throt=512: 746K ok/sec, 0 err (hardware ceiling) uring CT=8 t=16 throt=512: 743K ok/sec, 0 err (hardware ceiling) libaio CT=1 t=32 throt=120: 636K ok/sec, 0 err (production default) uring CT=8 t=32 throt=120: 618K ok/sec, 0 err (production default) * Tsavorite Native: make Initialize idempotent via lazy native-handle creation NativeStorageDevice.Initialize used to perform all the heavy work (native device creation, completion-thread spawn, ABI / segment-size / sector-size cross-checks) eagerly and threw "called more than once" on a second call. The other IDevice implementations (LocalStorageDevice, RandomAccessLocalStorageDevice, ManagedLocalStorageDevice) all inherit a metadata-only StorageDeviceBase.Initialize that simply overwrites segmentSize / segmentSizeBits / mask fields and is silently idempotent. They open their per-segment OS handles lazily inside the IO methods. This contract mismatch broke any caller that invokes Initialize twice on the same NSD instance. The canonical case is LocalStorageNamedDeviceFactory.Get(), which calls Initialize(-1L) as a defensive pre-init so consumers can't forget; the consumer (snapshot checkpoint state machine SnapshotCheckpointSMTask, cluster checkpoint streaming TsavoriteCheckpointReader.CreateCheckpointDevice) then calls Initialize(actualSegmentSize). Under the old NSD that throws; under the new NSD it works the same way the other backends do. Implementation: - NSD.Initialize is now metadata-only — delegates to base.Initialize. Pre-flight argument validation (segmentSize power-of-two, sector-size floor, omitSegmentIdFromFilename) is preserved. - New EnsureNativeDeviceCreated() does the heavy work, lazily, on first IO. Reads the latest base.segmentSize and base.OmitSegmentIdFromFileName so whichever Initialize call ran most recently wins. - Thread-safe via double-checked locking on a new nativeCreateLock. The publish of nativeDevice uses Volatile.Write so a second observer of nativeDevice != IntPtr.Zero is guaranteed to see a fully-initialised handle with completion threads already running. - Dispose now also takes nativeCreateLock around the cancel-join-destroy sequence so it cannot race with a concurrent EnsureNativeDeviceCreated (which would otherwise leak a freshly-published native handle and its completion threads). - IO entry points (ReadAsync, WriteAsync) call EnsureNativeDeviceCreated() before submission. Bookkeeping entry points (Reset, TryComplete, GetFileSize, RemoveSegment) no-op when the native handle has not been created yet, matching the semantics of the other backends (Reset on a device with no open handles is a no-op). Verified against the full unit-test sweep with Native forced as the default device (the GetDefaultDeviceType hack is local-only and not in this commit): Tsavorite.test: 206 / 206 (was 204 / 206 pre-fix) Garnet.test: 789 / 789 (was 110 / 792 pre-fix; 681 were blocked on Initialize-twice) Garnet.test.acl: 425 / 425 Garnet.test.collections: 746 / 746 Garnet.test.complexstring: 386 / 386 Garnet.test.rangeindex: 62 / 62 Garnet.test.vectorset: 42 / 42 No change to behavior for callers that invoke Initialize once with the real segment size, which is what every production code path already does. * Tsavorite Native: probe GetFileSize/RemoveSegment without forcing native handle GetFileSize and RemoveSegment must report the on-disk state regardless of whether IO has flowed through the device, matching LocalStorageDevice and RandomAccessLocalStorageDevice semantics. Before this fix, both no-op'd when no native handle had been created — which silently truncated the cluster manager's recovery decision because ClusterManager.cs:79 and ReplicationManager.cs:160 call `device.GetFileSize(0) > 0` to decide whether to recover persisted cluster config / replication history. With Native, a restarted node would always "Initialize new node instance config" instead of recovering, get a fresh node ID, and fail every replication-resume test (e.g. ClusterSRNoCheckpointRestartSecondary which restarts a replica and then waits for AOF sync to catch up). Changes: * GetFileSize now falls back to FileInfo when no native handle exists — same shape as RandomAccessLocalStorageDevice.GetFileSize (open-on-demand) but without paying io_uring/libaio setup cost just to stat a file. * RemoveSegment now falls back to File.Delete when no native handle exists — same shape as LocalStorageDevice / RandomAccessLocalStorageDevice (best-effort unlink, swallows ENOENT). * Per IDevice contract enforced in 889def4 ("Tsavorite IDevice: unify Initialize contract — required for all devices, -1 = unbounded single segment"), ReadAsync / WriteAsync now call EnsureInitialized() before EnsureNativeDeviceCreated() so the IDevice_*BeforeInitialize_Throws hardening tests get the same InvalidOperationException shape from Native that they get from the other devices. * Two device tests updated to match the lazy-Initialize contract that was introduced in commit f4e3044 ("Tsavorite Native: make Initialize idempotent via lazy native-handle creation"): - NativeStorageDevice_InitializeTwice_Throws → _Idempotent: idempotent Initialize matches the LSD/RA contract used by LocalStorageNamedDeviceFactory.Get + consumer re-init pattern. - NativeStorageDevice_Recovery_LargerExistingSegment_DetectsMismatch: the C++ ValidateRecoveredSegments check now fires on first IO (when EnsureNativeDeviceCreated runs), not at Initialize time, so the test asserts on a ReadAsync rather than Initialize. Verified on Linux x64 / .NET 10: * libs/storage/Tsavorite/cs/test/test.hlog: all IDevice + NativeStorageDevice tests pass (62/62) with Native default * test/cluster/Garnet.test.cluster.replication: all 4 ClusterSRNoCheckpointRestartSecondary variants pass with Native default (regression-test for the recovery path) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: wake completion drainer on Dispose via no-op IO Without this fix, NSD.Dispose() can stall up to CompletionWorkerTimeoutSecs (1s) per io_context because the completion-drainer thread is blocked in io_getevents / io_uring_wait_cqe_timeout waiting for events that will never come (the IO drain phase has already brought numPending to 0). The Thread.Join following completionThreadToken.Cancel() then has to wait for the next QueueRunFor timeout to fire so the thread can observe cancellation and exit. This was visible as exactly-1.0s gaps in cluster replication recovery traces: checkpoint metadata reads / writes that each create+dispose a fresh NSD spent ~1s in Dispose, multiplying across the ~5–10 devices created per checkpoint into multi-second stalls. ClusterReplicaSyncTimeoutTest (replicaSyncTimeout=1s) and MultiDatabaseSaveRecoverByDbIdTest(True) (2s LASTSAVE poll window) failed because of this; the actual I/O on Native is microseconds, not seconds. Fix: post a synthetic wake-up event on each io_context when Dispose runs. * libaio: submit a 0-byte read on a /dev/null fd opened in the handler ctor. /dev/null completes immediately and does not require O_DIRECT alignment, so the wake-up does not interfere with the real segment files. * io_uring: submit io_uring_prep_nop with user_data = nullptr; the drain loop recognises nullptr as a wake-up sentinel and skips dispatch. * Windows ThreadPoolIoHandler has no dedicated drainer (callbacks fire on threadpool threads), so its Wake is a no-op stub returning 0. The completion thread wakes from its blocking syscall almost immediately, observes the cancellation token on its next loop iteration, and exits. No extra idle work, no polling, no shortened timeout. * NSD.Dispose latency: 1025ms worst case -> ~20-30ms (microbenchmark). * ClusterReplicaSyncTimeoutTest with Native: ~22-25s (fail) -> ~3s (pass). * MultiDatabaseSaveRecoverByDbIdTest(True) with Native: timeout (fail) -> ~6s (pass). * Idle drainer syscall rate is unchanged (1/s/context). C ABI changes (additive — old exports preserved): * NativeDevice_WakeCompletionWorker(device, ctx_idx). * INativeDevice::Wake; QueueIoHandler::Wake, UringIoHandler::Wake, ThreadPoolIoHandler::Wake stub. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite IDevice: make Initialize() optional — ctor defaults are valid for IO Background: commit 889def4 ("unify Initialize contract — required for all devices") added an EnsureInitialized() guard that threw InvalidOperationException at every IO entry point if Initialize() had not been called first. This was redundant: the ctor already establishes segmentSize=-1 / segmentSizeBits=64 / segmentSizeMask=~0UL, which is functionally identical to having called Initialize(-1) — every absolute address right-shifts to segment 0, producing unbounded single-segment routing. The mandatory-Initialize contract was the root cause of the entire factory-pre-init + NSD lazy-creation saga: LocalStorageNamedDeviceFactory.Get was forced to call device.Initialize(-1L) defensively just to satisfy the contract, which broke NativeStorageDevice (its single-shot Initialize then asserted on the consumer's follow-up Initialize(realSize)). Recent commits f4e3044 + 0535da0 papered over this with lazy native-handle creation; this commit removes the root cause. Changes: * StorageDeviceBase: remove the 'initialized' flag, EnsureInitialized() helper, and ThrowNotInitialized() method. Initialize() is now purely a *configuration* call to override the ctor defaults (set a non-default segment size, opt into OmitSegmentIdFromFileName). The ctor doc explicitly states that callers may issue IO immediately after construction. * All concrete devices: remove the EnsureInitialized() calls at the top of ReadAsync / WriteAsync / TruncateUntilSegmentAsync / RemoveSegment (libaio, io_uring, RA, ManagedLocal, LocalMemory, Null, Tiered, Sharded, Azure). * LocalStorageNamedDeviceFactory.Get: drop the defensive device.Initialize(-1L); the ctor defaults match what that call did anyway. * NSD's recent EnsureInitialized() additions to ReadAsync/WriteAsync (introduced in 0535da0 only to satisfy the hardening test) are also removed by the sweep. * ComponentRecoveryTests.cs: drop 3 redundant Initialize(-1L) calls. * test.hlog DeviceTests: rename and repurpose IDevice_*BeforeInitialize_Throws to IDevice_*BeforeInitialize_UsesCtorDefaults — the new test demonstrates that WriteAsync/ReadAsync on a freshly-constructed device (no Initialize call) works correctly using the unbounded single-segment defaults, across Native / RA / ManagedLocal. TestUtils.cs:165 still calls device.Initialize() — that path is conditional on the caller wanting OmitSegmentIdFromFileName=true, which IS only settable via Initialize (it is not a ctor parameter), so the call is genuinely needed there. Verified on Linux x64 / .NET 10: * libs/storage/Tsavorite/cs/test/test.hlog (IDevice + NativeStorageDevice tests): 62/62 pass. * libs/storage/Tsavorite/cs/test/test.recovery (ComponentRecovery tests): 4/4 pass. * Full Garnet.test, Garnet.test.cluster, Garnet.test.acl, Garnet.test.collections, Garnet.test.extensions, Garnet.test.scripting, Garnet.test.complexstring, Tsavorite IDevice+NSD: pass at the same rates as before this commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native + IDevice: refresh comments to reflect final design Sweep the PR for comments that referenced earlier design choices as they evolved during development, and rewrite them to describe the steady-state contract directly without historical baggage. * IDevice.Initialize: replace the "Must be called exactly once... before any IO entry point... uninitialized device throws InvalidOperationException" doc with the actual contract: Initialize is purely an opt-in configuration step to override the ctor defaults (which are equivalent to Initialize(-1)); callers may issue IO immediately after construction. * NativeStorageDevice ctor doc: rewrite to describe the as-shipped lazy creation flow (configuration captured at ctor, native handle created on first IO via EnsureNativeDeviceCreated) rather than the stale "Native device creation is DEFERRED until Initialize... every IO entry point throws InvalidOperationException" framing. * NativeStorageDevice.Initialize doc: drop the misleading "Creates the underlying native device with the requested segment size" lead-in (which hasn't been true since the lazy-creation refactor); replace the "factory pre-init" example (factory no longer pre-inits) with a steady-state description of when repeat Initialize calls are honoured. * NativeStorageDevice.EnsureNativeDeviceCreated doc: replace "Throws if Initialize has not been called" (now uses ctor defaults if no Initialize) with "Throws if the device has been disposed or if the native shim rejects the configuration". * NativeStorageDevice.EnsureReadyOrSilent doc: drop the "does not throw on 'not initialized yet'" qualification. * NativeStorageDevice.GetSectorSize doc: re-point the "cross-check" link from Initialize to EnsureNativeDeviceCreated (which is where it actually happens). * NativeStorageDevice.Dispose doc + body: bound the worst-case shutdown stall by the longest in-flight user callback (not CompletionWorkerTimeoutSecs) since wake-up uses NativeDevice_WakeCompletionWorker; rewrite the inline Dispose comment so it documents the steady-state design rather than what it improved over. * NativeStorageDevice nativeSegmentSizeBytes / UnboundedNativeSegmentSizeBytes field doc: clarify that the value is populated by EnsureNativeDeviceCreated (not Initialize) and that the default is reached without calling Initialize. * NativeStorageDevice_InitializeTwice_Idempotent test: drop the now-stale "factory pre-init... consumer re-initializes" rationale; describe the idempotent contract directly. * NativeStorageDevice_DisposeBeforeInitialize_IsNoOp test: drop the "Phase 6" reference and reword in terms of the steady-state lazy-creation contract. * SimulatedFlakyDevice.Initialize: replace the "so its EnsureInitialized() guard passes when our IO methods delegate to it" comment (the guard no longer exists) with a description of why both devices need matching geometry. * LinuxFileExtensions.OpenDirect dsync param doc: drop the "previously asked for it" wording — the WriteThrough callsites still pass it; describe the parameter as an opt-in for WriteThrough-equivalent semantics. * Doc-cref bookkeeping: change <see cref="base.segmentSize"/> (illegal cref for inherited fields) to <c>base.segmentSize</c> code spans. No behaviour change. Build clean on Garnet.slnx and Tsavorite.slnx; dotnet format --verify-no-changes clean on both. IDevice + NSD device tests all pass (62/62). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: ship libnative_device.so without liburing dependency Background: the shipped libnative_device.so was built with -DUSE_URING=ON, so it had a hard DT_NEEDED entry for liburing.so.2. Loading the .so on a host without liburing2 installed (e.g. a GitHub Actions ubuntu-latest runner, or an end-user box where only libaio is in the base image) failed with: System.DllNotFoundException: ... liburing.so.2: cannot open shared object file: No such file or directory …even for callers that only ever requested the libaio backend, because the dynamic linker resolves NEEDED libraries at load time regardless of which exported symbols the caller goes on to invoke. Rebuild the prebuilt with -DUSE_URING=OFF so the shipped .so links only libaio. Most Linux distributions ship libaio in the base system, so the prebuilt now loads without any additional setup. The io_uring backend becomes a build-time opt-in: callers that want it install liburing-dev and rebuild with -DUSE_URING=ON. The C# layer already surfaces a clear TsavoriteException for callers that request Uring against a USE_URING=OFF build ("Requested IO backend 'Uring' is not available in the loaded native_device library… Rebuild the native library with -DUSE_URING=ON and install liburing-dev to enable io_uring."). Build fix: file_linux.h now includes <fcntl.h> directly (for the ::open() / O_RDONLY usage in QueueIoHandler::OpenWakeFd()). Previously these were pulled in transitively through <liburing.h>, which is now gated behind #ifdef FASTER_URING. Dockerfile updates: drop liburing2 / liburing from the runtime install list in all 5 Dockerfiles (default, .ubuntu, .alpine, .azurelinux, .chiseled). Comments left for users that rebuild with USE_URING=ON. README updates: rewrite the "Runtime dependencies" section to describe the new default (libaio only). Replace the "Disabling io_uring (optional)" section with "Enabling io_uring (optional)". Verified on Linux x64 / .NET 10: libaio default works (62/62 IDevice + NativeStorageDevice tests pass); ldd confirms only libaio.so.1t64 is in NEEDED. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: ship two .so flavors — uring-enabled + libaio-only fallback Single-shipping libnative_device.so created a deployment dilemma: build with USE_URING=ON and end-users without liburing get DllNotFoundException at load time; build with USE_URING=OFF and the io_uring backend stops working even on hosts that DO have liburing installed (which is the case where uring matters for perf — modern NVMe at >1M IOPS benefits noticeably from uring over libaio). Ship both flavors instead: * libnative_device.so — built with USE_URING=ON; DT_NEEDED on libaio AND liburing. Exposes both Libaio and Uring backends. * libnative_device_libaio.so — built with USE_URING=OFF; DT_NEEDED on libaio only. Exposes the Libaio backend. NativeStorageDevice's DllImportResolver tries the uring-enabled binary first; on DllNotFoundException matching 'liburing.so.2: cannot open' it falls back to the libaio-only binary. The Libaio backend therefore always works out of the box on any Linux distribution that ships libaio (essentially all of them). liburing is opt-in: hosts that install it get the Uring backend with zero runtime overhead vs Libaio (direct calls, no function-pointer indirection — we deliberately rejected the dlopen approach so the future-default uring path stays optimal). If a caller explicitly selects IoBackend.Uring on a host without liburing, the construction-time error message now points at the install command per distro ('apt-get install -y liburing2', 'dnf install -y liburing', 'apk add liburing') instead of telling the user to rebuild the .so with -DUSE_URING=ON. We never silently downgrade Uring to Libaio. Changes: * NativeStorageDevice.cs: new LibaioFallbackLibraryPath; ImportResolver catches DllNotFoundException for liburing.so.2 and falls back to the libaio-only .so. ResolveNativeLibraryPath now takes the path as a parameter so it can resolve either flavor. * NativeStorageDevice.cs: rewrite the 'backend not available' exception message — point at install commands (the actual remediation) not rebuild. * Tsavorite.core.csproj: add libnative_device_libaio.so as a second ContentWithTargetPath asset so both .so files are copied to the output directory and packed into the NuGet runtime payload. * runtimes/linux-x64/native/libnative_device.so — REPLACED with USE_URING=ON build (DT_NEEDED libaio + liburing). 2.3 MB. * runtimes/linux-x64/native/libnative_device_libaio.so — NEW, USE_URING=OFF build (DT_NEEDED libaio only). 1.6 MB. * Dockerfile, Dockerfile.ubuntu, Dockerfile.alpine, Dockerfile.azurelinux, Dockerfile.chiseled: re-add liburing2 / liburing to the runtime installs so docker users get the io_uring backend out of the box (the libaio-only fallback would otherwise leave Uring unusable inside containers). * cc/README.md: rewrite the 'Runtime dependencies' and build sections to describe the two-flavor layout, drop the stale 'Enabling io_uring' section, and document the prebuilt rebuild workflow. Verified end-to-end: * Both backends saturate the Dell P5600 NVMe at ~743K random read IOPS in benchmark/Device.benchmark (matches the pre-change reference). * 62/62 IDevice + NativeStorageDevice tests pass. * dotnet format clean on both Garnet.slnx and Tsavorite.slnx. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native (Windows): add init_errno()/initialized() stubs to ThreadPoolIoHandler NativeDeviceImpl's constructor (native_device.h:100-101) gates on handler_.init_errno() to surface an actionable error message when the underlying IO handler failed to initialize (e.g., libaio io_setup() failed with EMFILE / ENOMEM, or io_uring_queue_init() failed). The Linux handlers QueueIoHandler and UringIoHandler both expose this API; ThreadPoolIoHandler (Windows) did not, so MSVC failed to instantiate NativeDeviceImpl<ThreadPoolIoHandler> with: error C2039: 'init_errno': is not a member of 'FASTER::environment::ThreadPoolIoHandler' Add init_errno() and initialized() stubs that return 0 / true unconditionally — the Windows ThreadPool API does not have a separable init step that can fail in the same way the Linux io_setup / io_uring_queue_init paths can (threadpool creation failures propagate via threadpool_'s ctor, not via a later 'check this' field on the handler), so the stubs are semantically correct. NativeDeviceImpl then falls through to the log_.Open(&handler_) path which is where Windows-specific errors (missing directory, permission denied, etc.) actually surface. Linux unaffected: rebuilt build/Release-uring cleanly after this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native tests: stop skipping NSD tests on Windows The Native tests in DeviceTests.cs had blanket 'NativeStorageDevice is Linux-only' Assert.Ignore guards that dated from when NSD's C++ shim was Linux-only. The shim is built on Windows too (native_device.dll via the ThreadPool / IOCP backend in file_windows.cc), so directly constructing 'new NativeStorageDevice(...)' works on Windows. The blanket guards were silently dropping ~15 NSD test cases on Windows CI. Drop the guards so the tests exercise the Windows C++ shim. The legitimate Linux-only guard on IDevice_PermissionDeniedAtFirstWrite_CallbackGetsError (chmod-based; chmod has no Windows analogue) is preserved. Important: end-user device routing is UNCHANGED. Devices.CreateLogDevice(DeviceType.Native) on Windows still returns LocalStorageDevice (managed Windows IOCP), not NativeStorageDevice — that routing happens in Devices.cs and was not touched. These tests directly instantiate the NSD class for shim-coverage purposes only; they do not affect what end users get from the default device factory. Linux: 62/62 pass after this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: rebuild win-x64 native_device.dll for the latest C++ source Rebuild the shipped Windows prebuilt from the current native_device source so the latest fixes (Initialize idempotence, WakeCompletionWorker, etc.) are reflected in the win-x64 DLL. USE_URING is a no-op on Windows; the DLL only exposes the Default (IOCP) backend, so there is no equivalent of the libnative_device_libaio.so fallback on this platform. End-user device routing is unchanged: Devices.CreateLogDevice(DeviceType.Native) on Windows still returns LocalStorageDevice (managed Windows IOCP). This DLL is exercised by direct 'new NativeStorageDevice(...)' construction (see Tsavorite.test.hlog DeviceTests — 59/59 Native + IDevice tests pass on Windows after this rebuild). Built with: Visual Studio 17 2022, MSVC v143, x64, Release configuration, Spectre-mitigated CRT. * Tsavorite Native: address GPT-5.5 PR review findings Fixes two correctness issues caught by an automated code review of the optimize-device PR. ### io_uring SQE leak on submit failure In UringFile::ScheduleOperation, io_uring_get_sqe() advances the user-side sqe_tail before io_uring_submit() is called. If submit fails after retries (-EAGAIN/-EBUSY exhausted, or any other negative), the old code released the lock and returned IOError without doing anything about the still-pending SQE. user_data on that SQE pointed at the io_context unique_ptr that was about to be freed by the guards unwinding, so the next successful submit on the same ring would consume the stale SQE and the QueueRunFor drain loop would dispatch a callback against freed memory — a clear use-after-free. Fix: before releasing sq_lock on the failure path, rewrite the still- pending SQE in place as io_uring_prep_nop with user_data = nullptr. The drain loop already skips nullptr user_data (it's the wake-up sentinel used by UringIoHandler::Wake), so when a later submit flushes this nop the CQE is drained harmlessly. Safe to mutate the SQE in place because we still hold sq_lock and no kernel/concurrent submitter has observed it yet. ### NativeDevice sector_size always returned 512 FileSystemSegmentedFile::alignment() returned a hard-coded 512. NativeDeviceImpl::sector_size() delegated to it, so the C# wrapper's sector-size cross-check in EnsureNativeDeviceCreated would: - falsely throw on 4K-native disks where ProbeAlignment returns 4096 (managed 4096 vs native 512 → 'sector-size mismatch' → device unusable), or - on 4K-native disks where the managed probe fell back to 512 (e.g. older kernel without STATX_DIOALIGN), let the device initialize with SectorSize=512 and then have the kernel reject the 512-aligned O_DIRECT buffers with EINVAL. Fix: factor the STATX_DIOALIGN probe from NativeDevice_ProbeAlignment into a shared inline helper (native_device::ProbeDioAlignment in native_device.h) and call it once from the NativeDeviceImpl ctor, caching the result as the immutable member device_alignment_. sector_size() now returns the cached value; NativeDevice_ProbeAlignment delegates to the same helper. Both sides of the ABI cross-check go through identical probe logic, so the check is now a meaningful ABI / runtime-drift detector instead of a 4K-disk footgun. ### Stale IDevice.Initialize XML The omitSegmentIdFromFilename param said it was 'only supported by managed devices — NativeStorageDevice rejects this flag'. Native devices have honored the flag since 6584cf7. Updated the doc. ### Alpine install hint The 'IoBackend.Uring with libaio fallback' error message suggested 'sudo apk add liburing' on Alpine, but README.md notes that the prebuilt won't load on Alpine (musl) at all. Replaced the apk suggestion with the actual Alpine support story (use a glibc image or fall back to a managed device). Verification: 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass on Linux. Both .so binaries rebuilt (uring-enabled and libaio-only fallback) with correct ldd output. Device.benchmark NVMe saturation throughput unchanged within noise (libaio 738K IOPS, uring 349K IOPS on Dell P5600). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: probe sector size via sysfs max(logical, physical) Replaces the statx(STATX_DIOALIGN) probe in ProbeDioAlignment with a direct sysfs lookup of max(logical_block_size, physical_block_size). Why: - STATX_DIOALIGN reports only the kernel-enforced minimum (= logical block size). It misses the firmware's preferred sector (physical_block_size), so on a 512e drive (logical=512, physical=4096) the probe would return 512 and Tsavorite would take a firmware RMW penalty on every partial-sector write. - STATX_DIOALIGN also requires kernel 6.1+ AND the filesystem to populate the field; ext4 on 6.8 leaves it unset on 512-byte devices, so the probe was already falling through to the 512 default in practice. - sysfs gives us both values directly, on every kernel, with no O_DIRECT dance. Taking max(logical, physical) covers the correctness floor (logical = kernel-enforced minimum) and the performance floor (physical = avoid RMW on partial writes) in one shot. Implementation: - stat() the file (or its closest existing ancestor — log file may not exist yet at construction). Extract st_dev → (major, minor). - Read /sys/dev/block/<maj>:<min>/queue/{logical,physical}_block_size. For partitions (e.g. sda2), the queue/ dir lives on the parent whole-disk block device — fall through to ../queue/<field>. - Round result up to a power of two (always already pow2 on real hardware) and floor at 512 B. On this machine (Dell P5600 NVMe + PERC sda): Probe(/DATA2/badrishc) = 512 (NVMe: logical=512, physical=512) Probe(/tmp/devbench) = 512 (sda partition via parent-walk) Probe(/home/badrishc) = 512 Probe(/) = 512 All values match max(logical, physical) read directly from sysfs. Verification: - 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass - Both .so flavors rebuilt (uring-enabled + libaio-only fallback) - C ABI NativeDevice_ProbeAlignment delegates to the same helper, so managed SectorSize and native sector_size() remain in lockstep. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native tests: align test buffers to 4096, matching sysfs-probed sector size CI failure on IDevice_Initialize_OmitSegmentIdFromFilename_BareFileName (and likely other IDevice_* tests on the same runner) reported: 'NativeStorageDevice.WriteAsync: misaligned I/O — sector size is 4096, but offset=0x0, length=4096, buffer=0x...7EC7A9F76800' The buffer ends at 0x...800 = 2048 — i.e. 2048-aligned but not 4096-aligned. The test helper allocated buffers aligned to HardeningSectorSize = 512 (the pre-PR default for every Garnet Linux device); a 512-aligned formula can land on a 2048-boundary that is not also a 4096-boundary. CI's underlying disk reports physical_block_size = 4096 in sysfs, so the new max(logical, physical) probe returns 4096 there. The native shim then correctly rejects sub-4096-aligned O_DIRECT buffers with EINVAL. The fix is on the test side: bump HardeningSectorSize from 512 to 4096 so the test buffer alignment matches the strictest device.SectorSize seen on any modern hardware (512n, 512e, 4Kn). Locally (Dell P5600 NVMe, logical=physical=512 → SectorSize=512) all 62 IDevice + NativeStorageDevice tests still pass — 4096 trivially divides 512. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite: revert GetAndPopulateReadBuffer changes (defer to separate PR) The read-window sizing optimization (drop leading-slop padding + clamp to page-end) is out of scope for this device-backend PR; it interacts with the larger read-IO path and deserves its own focused PR with dedicated benchmarking. Reverting to the pre-PR behavior here. TryAllocateRetryNow's bounded-backoff change is retained — it's a self-contained allocator hot-path fix and is independently verified (+13.9% on YCSB load with libaio at 64 threads, per kvbench benchmarking). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: Windows probe uses IOCTL_STORAGE_QUERY_PROPERTY for max(logical, physical) Symmetry with the Linux sysfs probe — Windows now reads both BytesPerLogicalSector and BytesPerPhysicalSector from the volume's STORAGE_ACCESS_ALIGNMENT_DESCRIPTOR (via IOCTL_STORAGE_QUERY_PROPERTY + StorageAccessAlignmentProperty) and returns the rounded-up-to-pow2 max, floor 512 B. Previously the Windows branch returned 512 unconditionally, which would silently undersize SectorSize on Windows 4Kn / 512e drives. Implementation: - Parse drive letter from filename ("C:\foo.dat" -> "\\.\\C:"). UNC paths are not supported by this probe — fall back to 512. - CreateFile on the volume with FILE_READ_ATTRIBUTES (no admin needed). - DeviceIoControl(IOCTL_STORAGE_QUERY_PROPERTY) with StorageAccessAlignmentProperty. - max(logical, physical), round up to pow2, floor 512. Linux behavior unchanged. Both .so flavors rebuilt and pass 62/62 device tests on this machine (logical=physical=512 NVMe). REQUIRES Windows DLL rebuild — the Windows path in ProbeDioAlignment is now non-trivial, and the existing prebuilt native_device.dll still returns 512 unconditionally. Without the rebuild, on a Windows 4Kn box the managed SectorSize cross-check would (incorrectly) pass at 512 while the device might actually need 4096. Rebuild recipe in the companion review comment / Tsavorite/cc/README.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add binaries * Tsavorite Native: skip wake-up/failed-submit sentinel CQEs in TryCompleteFor Addresses Copilot review comment on file_linux.cc:373. UringIoHandler::TryCompleteFor (and via it TryComplete) dispatched every drained CQE through DispatchUringCqe without checking the user_data = nullptr sentinel that QueueRunFor already handles. The sentinel marks two kinds of no-op CQEs: - Wake-up nops submitted by UringIoHandler::Wake to unblock the drainer on Dispose. - SQEs rewritten in-place after io_uring_submit failed (the SQE leak fix in c6d68925); these are committed to the SQ but carry no caller context. If a TryComplete() / TryCompleteFor() call picks up either kind of nop CQE, DispatchUringCqe would dereference the null context at context->callback(...) and segfault. Fix: mirror the nullptr-skip from QueueRunFor in TryCompleteFor. Return true to count the drain (matching the any-flag semantics). Verification: 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass. uring .so rebuilt; libaio-only .so is byte-identical because the patched code path is wrapped in #ifdef FASTER_URING and not compiled into the libaio-only fallback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native (uring): batch-drain CQEs and dispatch outside cq_lock QueueRunFor used to acquire cq_lock per CQE (peek -> read fields -> cqe_seen -> release -> dispatch). With a single drainer thread that serializes lock acquire/release on every completion and forces submitters to wait through callback latency when they need ring access. Replaced with the canonical liburing batch-drain idiom: - acquire cq_lock once - io_uring_peek_batch_cqe(ring, cqes, 64) to pull up to 64 CQEs - snapshot (io_res, context) for each - io_uring_cq_advance(ring, n) to release the slots - release cq_lock - dispatch callbacks outside the lock This is the io_uring equivalent of libaio's io_getevents(n) per syscall. Snapshot BEFORE cq_advance is mandatory because the kernel may reuse CQ slots once advanced, leaving the cqe pointers dangling. The wake-up / failed-submit sentinel (user_data == nullptr) is still skipped without dispatch, same as before. Measured impact on Dell P5600 (16 submitter threads, batch 64, throttle 256): ct=1 (1 ring, 1 drainer): 339K -> 354K ops/sec (+4%) ct=4 (4 rings, 4 drainers): 735K -> 737K (saturates, noise) ct=8 (8 rings, 8 drainers): 750K -> 742K avg (saturates, noise) The single-drainer gain is modest because the real bottleneck at ct=1 with 16 submitters is sq_lock contention on the single ring, not cq_lock contention. The batch-drain is still strictly better: - dispatches outside the lock so submitters aren't blocked by user-callback latency, - matches the idiomatic liburing pattern, - amortizes the lock acquire/release across up to 64 CQEs per cycle. For high-throughput workloads, sharding across multiple rings remains the right scaling lever (ct >= 4 saturates this drive). Verification: 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass. uring .so rebuilt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native (uring): per-thread ring affinity + 4 default rings Eliminates the sq_lock contention that was capping uring at ~340K IOPS at the default numCompletionThreads=1. Two changes work together: 1. Per-thread ring affinity in pick_ring (file_linux.h): Each submitter thread is assigned a ring on its first submit (round- robin against other threads via an atomic counter) and keeps that assignment for life. Same-thread submits never contend on sq_lock with themselves; different threads only contend when they got assigned the same ring (num_submitter_threads > num_rings). This is the user-space equivalent of libaio's "io_submit is thread-safe per io_context" — eliminate shared mutable state across submitters. 2. Hardcoded 4 rings for uring (NativeStorageDevice.cs): numIoContextsConfig = ioBackend == Uring ? max(kDefaultUringRings=4, numCompletionThreads) : numCompletionThreads So uring always has at least 4 rings even at numCompletionThreads=1. The single drainer covers all 4 rings via the legacy QueueRun compat scanner (CompletionWorker passes ctxIdx=-1 in that case). libaio is unchanged: rings == numCompletionThreads (extra rings don't help; the kernel io_context mutex is already efficient). Result on Dell P5600 NVMe (16 submitter threads, batch 64, throttle 256): Before (1 ring, 1 drainer): ~340K After (4 rings, 1 drainer, default): ~700K (matches libaio ct=1) After (8 rings, 8 drainers, sharded): ~745K (unchanged, was already saturating) No new public configuration parameters. numCompletionThreads still controls drainer count; the ring count is now backend-derived behind the scenes. The CompletionWorker single-drainer-multi-ring path was added specifically so the default numCompletionThreads=1 case can saturate without spawning extra drainer threads. Also: bumped HardeningSectorSize and the legacy bufferPool / NativeDeviceTest2 sector_size constants from 512 to 4096 to match the strictest device SectorSize we expect on any modern hardware (4Kn drives where the new max(logical, physical) probe returns 4096). Tests would otherwise fail with EINVAL on 4Kn CI runners with 512-aligned buffers. Verification: - 64/64 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass - Default uring (no flags) hits 626-740K across t=1..64 vs ~340K before - Sharded ct=4/8 unchanged (still saturates) - libaio default unchanged Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add binaries * Tsavorite Native tests: fix ReadInto length mismatch surfaced by 4096 SectorSize NativeDeviceTest1 read 1024 bytes (entryLength) using ReadInto, which: - rounded the read length up to the device sector size, - then returned a buffer of that ROUNDED length, - which the caller compared via SequenceEqual against the original `entry` byte[] (length 1024). When SectorSize was 512 (the old constant probe), 1024 rounded to 1024 and the lengths happened to match. With the new max(logical, physical) probe returning 4096 on 4Kn drives (Windows/Ubuntu CI runners), 1024 rounds to 4096, the returned buffer is 4096 bytes long, and SequenceEqual fails on length mismatch (regardless of content). Pre-existing latent bug — the rounding to sector size is correct for the IO submit, but the caller should only see the bytes it asked for. Fix: return a buffer of the caller-requested logical `size`, not the sector-rounded `numBytesToRead`. Verification: 64/64 Tsavorite.test.hlog NativeDeviceTest + IDevice + NativeStorageDevice tests pass on Linux (where SectorSize is 4096 on the CI runner's 4Kn drive). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite benchmarks: refresh stale completion-threads help text The --device-completion-threads (KV.benchmark) and --completion-threads (Device.benchmark) help text said "all drainers share the same kernel io_context / io_uring" and "values > 1 are rarely useful past 1 today". Both claims are stale since the sharded-rings work (8cbca9d4d) and the per-thread ring affinity + 4-default-rings change (298bfd180): - Each drainer is bound 1:1 to its own kernel io_context (libaio) or io_uring ring (uring). - Submitters distribute across rings via per-thread affinity. - For io_uring, throughput scales with completion-threads up to available submitter concurrency (measured: ct=1 ~340K → ct=4 ~735K on Dell P5600 NVMe at the device-benchmark level). - For libaio extra drainers still rarely help past 1 (kernel per-context mutex is efficient). - Note added that uring uses min 4 rings even at ct=1 with the single drainer covering all rings via the legacy QueueRun scanner. Help-text-only change. No code behavior change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: fix KV.benchmark deadlock on multi-segment disk-spill reads Root cause: cross-segment read rejection + engine retry loop ============================================================== The AllocatorBase.GetAndPopulateReadBuffer sector-aligned read window can extend past the page-end boundary when reading a record near the tail of a page. When the device's segment size is a multiple of the page size (e.g. 4MB pages, 1GB segments — the Garnet default), an over-extended read at the last page of a segment also crosses the device's segment boundary. NativeStorageDevice's underlying FileSystemSegmentedFile rejects cross-segment reads with Status::IOError; the engine's AsyncGetFromDiskCallback interprets a 0-byte read as a short read and retries the same address — forever. Worker thread spins at 99% CPU, disk activity drops to zero, benchmark deadlocks. Reproduced reliably on KV.benchmark: --device native --device-io-backend libaio \ --log-memory 16m --page-size 4m --segment-size 1g \ -n 10000000 (1.28 GB dataset → crosses 1GB segment boundary) Smaller datasets (1M = 128MB, fits in 1 segment) work; larger ones hang. RandomAccess device works on all dataset sizes because its managed segmented-file wrapper doesn't reject cross-segment reads. Diagnostic captured the exact symptom: a read at sourceAddress 0x3FFFF600 (1,073,739,776 — 2,560 bytes before the 1GB segment boundary) with readLength 4608 (sector-aligned record window) extends to 0x40000C00 — 2,560 bytes into segment 1. Native rejects with Status::IOError, callback fires with numBytes=0, engine retries. Fix: clamp the aligned read length so it never crosses page-end. ============================================================ Added in AllocatorBase.GetAndPopulateReadBuffer: var pageEndInFile = (ulong)(AlignedPageSizeBytes * (GetPage(fromLogicalAddress) + 1)); if (alignedFileOffset + alignedReadLength > pageEndInFile) alignedReadLength = (uint)(pageEndInFile - alignedFileOffset); Records never span page boundaries (HandlePageOverflow guarantees), so the actual record is fully readable within the clamped window — available_bytes reflects what we actually got from disk, and the engine continues normally. pageEnd is sector-aligned (PageSizeBits >= sector size), so the clamped length stays sector-aligned. Also reverted the uring "min 4 rings even at ct=1" experiment ============================================================= The earlier "default 4 rings for uring regardless of ct" change was fundamentally broken: with per-thread submit affinity (pick_ring's thread_local index), submitters bound to rings 1-3 never get their completions drained because the single drainer blocks on ring 0 with a 1-second QueueRun timeout and only briefly polls the other rings between wake-ups. The result is ~50x throughput degradation on workloads where submitters land on rings != 0 (KV.benchmark load phase dropped from 2.5M ops/sec to 54K ops/sec at t=1). Reverted to the simple rule: rings == numCompletionThreads. For uring perf scaling, users set numCompletionThreads >= expected submitter concurrency; each ring is then continuously drained by its dedicated drainer thread. Defense-in-depth hardening ========================== - NativeStorageDevice._callback now catches ALL exceptions from the user callback (was: try/finally but exception propagated). A managed exception escaping back into native code across the C ABI boundary silently terminates the drainer thread; the next submitter then spins forever in device.Throttle(). Now the exception is logged and swallowed so the drainer survives. - NativeStorageDevice.CompletionWorker has the same try/catch around the whole drain loop as defense-in-depth against unrelated managed exceptions (P/Invoke marshalling, IntPtr.Zero races with Dispose, etc.). - file_linux.cc QueueFile::ScheduleOperation (libaio) and UringFile::ScheduleOperation (uring) now retry submit-side EAGAIN indefinitely with bounded backoff (64 sched_yields, then 1ms nanosleeps) instead of returning Status::IOError after 8 yields. Surfacing transient EAGAIN as a permanent error creates the same retry-loop pathology as the cross-segment-read bug above. EAGAIN is the kernel saying "ring is full, try later"; it's not a real error and must not be exposed to the engine. Verification ============ KV.benchmark, 100M keys × 100B, 16MB log (mostly disk-spill), 1 completion thread, 100% reads: libaio: t=1 135K ops/sec, t=4 400K, t=8 444K, t=16 445K, t=32 404K uring: t=1 124K ops/sec, t=4 244K, t=8 265K, t=16 278K, t=32 272K Both backends stable across the full thread × dataset sweep (previously native+libaio hung on any 10M+ dataset; native+uring hung on every config). 64/64 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> | 3 个月前 | |
Update docker files to latest (#1600) * update dockerfiles * Fix Docker build issues: libaio path, tdnf, and workflow image names - Dockerfile: Add libaio.so.1t64 -> libaio.so.1 symlink for Ubuntu 24.04 t64 compat - Dockerfile.ubuntu: Copy libaio.so.1t64 and create libaio.so.1 symlink (consistent with CI workaround and Dockerfile) - Dockerfile.cbl-mariner: Revert dnf back to tdnf (Azure Linux 3.0 uses tdnf) - docker-linux.yml: Update GHCR image names to match new base OS (jammy -> noble, cbl-mariner2.0 -> azurelinux3.0) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Alpine Lua SIGSEGV: remove glibc-compiled KeraLua liblua54.so The KeraLua NuGet package bundles a glibc-compiled liblua54.so that gets published to /app/liblua54.so. On Alpine (musl libc), .NET loads this glibc binary first (app dir takes priority), causing SIGSEGV on any Lua EVAL command. Remove it in the build stage so .NET falls through to the musl-compiled system Lua library via the runtime symlink. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add libaio to chiseled image for native device support The chiseled (distroless) image was missing libaio.so.1, causing --device-type Native to fail with 'libaio.so.1: cannot open shared object file'. Add libaio1t64 to the libs-builder stage and copy both the library and compat symlink into the final chiseled image. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Dockerfile.ubuntu libaio arm64 arch mismatch Remove the hardcoded --platform=linux/amd64 builder stage that always copied x86_64 libaio into the runtime image regardless of target arch. Install libaio1t64 directly in the runtime stage via apt-get (alongside liblua5.4-0) so the correct architecture library is installed, matching the approach used in the default Dockerfile. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Docker image validation test script Comprehensive Python script that verifies all 5 Linux Docker images: - Build all Dockerfiles - Basic server tests (PING, SET, GET) - Lua EVAL scripting - Default device persistence (all platforms incl. Alpine) - Native device persistence (glibc platforms) - Library resolution checks (libaio, liblua54, libnative_device) - Optional multi-platform buildx (amd64+arm64) Usage: python3 test/docker-tests/validate_docker_images.py [--skip-build] [--multiplatform] [--images ...] Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove dead libaio copy in chiseled prep-runtime stage The libaio files copied into prep-runtime's /usr/lib were never transferred to the final chiseled image (only /usr/share/dotnet/shared is copied from prep-runtime). The final image already gets libaio directly from libs-builder. Remove the unused copy/symlink steps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename Dockerfile.cbl-mariner to Dockerfile.azurelinux The runtime base image is now azurelinux3.0 and the CI publishes as -azurelinux3.0. Rename the Dockerfile to match and update the workflow matrix and test script references. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use native lua-libs package on Azure Linux instead of Ubuntu copy Azure Linux 3.0 ships lua-libs (providing /usr/lib/liblua-5.4.so) in the base runtime image. Use a symlink to this native library instead of copying a cross-distro binary from an Ubuntu builder stage. This avoids potential glibc ABI compatibility issues and removes an unnecessary build stage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use synchronous SAVE instead of BGSAVE+sleep in Docker tests SAVE blocks until the checkpoint completes and returns, eliminating the race condition from the sleep-based BGSAVE approach. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> | 6 个月前 | |
[Tsavorite] Add native Linux storage backend, harden NativeStorageDevice, refresh storage benchmarks (#1831) * Port device/IO changes from optimize-v2-io onto kv-bench All 31 non-benchmark files changed on optimize-v2-io (vs its branch base d3677cfaa) ported here. Backup tag: optimize-v2-io-prerebase-backup @ 3f41f2bdf. Scope: device/IO/native-backend ONLY. Includes: Tsavorite C++ native device: - io_uring backend + pluggable C ABI (file_linux.cc/h) - error model split (native_device_error.h) - file_system_disk + native_device.h updates - CMakeLists + README Tsavorite C# device: - NativeStorageDevice: IoBackend enum (Default, Libaio, Uring), completion threads, production-readiness pass - LinuxFileExtensions.cs: P/Invoke open() for true O_DIRECT - ManagedLocalStorageDevice + RandomAccessLocalStorageDevice: O_DIRECT wiring on Linux - Devices.cs: router updates for new device APIs Tsavorite allocator + utilities: - AllocatorBase: bounded backoff in TryAllocateRetryNow - CompletionEvent: Wait(TimeSpan) overload Tsavorite checkpoint management: - LocalStorageNamedDeviceFactory + Creator surface ioBackend + completionThreads parameters Tests: - DeviceTests.cs updated for new device APIs Garnet host: - --device-io-backend, --device-completion-threads flags - defaults.conf updated, GarnetServerOptions wiring Dockerfiles (all 5): install liburing alongside libaio. Note: YCSB.benchmark and KV.benchmark are not modified by this commit; KV.benchmark is the supported benchmark on this branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * KV.benchmark: add uring backend, fix --validate for disk-spill, doc liburing runtime dep - Options: --device-io-backend now accepts 'uring' (aliases: io_uring, iouring) in addition to libaio/default; help text + validation error list updated to match. - KV.benchmark README: device-backend table now describes the libaio vs uring split, with a runtime-install snippet for liburing across Debian/Ubuntu, Fedora/RHEL/AzureLinux, and Alpine, plus link to the full Tsavorite Native Device docs. New 'native + libaio' and 'native + uring' rows added to the cookbook for constrained-log large-dataset workloads. - Tsavorite Native Device README: new top-level 'Runtime dependencies (end users)' section listing the apt/dnf/apk install lines and how to fall back to the no-liburing variant. - KV.benchmark Validate: fix two bugs that surface when load and run use different thread counts and when the log spills to disk: 1) writerThread reconstruction now uses ResolvedLoadThreads (not Options.Threads which is the RUN count), so --load-threads N with --threads M != N validates correctly. 2) Reads of records below HeadAddress return Status.IsPending; the previous code counted these as misses. Validate now issues reads in batches of 256 and drains via CompletePendingWithOutputs, verifying each completed output against the per-thread pattern. Verified end-to-end with both backends: 4.6M × 100B × 8T × log=256m (~580MB dataset > 256MB log → forces disk spill) × 50R/50U × --validate: native + libaio → [validate] OK, run = 1.27 M ops/s native + uring → [validate] OK, run = 1.02 M ops/s 4.6M × 100B × 8T × log auto (fits) × 95R/5U × --validate: native + libaio → [validate] OK, run = 16.10 M ops/s native + uring → [validate] OK, run = 16.33 M ops/s Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite: replace MarkHandleAsAsync reflection hack with RandomAccess on SafeFileHandle Background: MarkHandleAsAsync used reflection to flip SafeFileHandle.IsAsync's non-public setter so a P/Invoke-opened O_DIRECT FD could be wrapped in 'new FileStream(handle, isAsync: true)' without throwing 'Handle does not support asynchronous operations'. The flag is non-public in .NET 8/10, the hack was fragile across future runtime versions, and on Linux IsAsync is a contract gate (no real overlapped I/O exists for files), so the lie bought us nothing beyond letting the FileStream constructor accept the handle. RandomAccessLocalStorageDevice — refactor: - StorageAccessContext.handle is now SafeFileHandle (was FileStream). - CreateRead/WriteHandle: * Linux + O_DIRECT capable: LinuxFileExtensions.OpenDirect -> raw SafeFileHandle, no FileStream wrap. Page-cache bypass via the O_DIRECT flag at open(2), exactly as before. * Otherwise (Windows; or Linux when filesystem rejects O_DIRECT): File.OpenHandle(path, ..., FileOptions.Asynchronous | cast FILE_FLAG_NO_BUFFERING). On Windows this gives the runtime IOCP-bound OVERLAPPED I/O; on Linux it's page-cached. - All I/O goes through RandomAccess.{Read,Write}Async(safeHandle, memory, offset). On Windows: true kernel async via IOCP. On Linux: pread/pwrite dispatched to ThreadPool (same as before). - GetFileSize uses RandomAccess.GetLength(handle). - SetFileSize uses RandomAccess.SetLength(handle, size). LinuxFileExtensions: - MarkHandleAsAsync and the IsAsyncProperty reflection are deleted entirely (no remaining callers). - System.Reflection using removed. ManagedLocalStorageDevice: - Reverted to origin/main. This device is designed to stay within FileStream APIs; the O_DIRECT branch we added doesn't belong here. Verified: - Tsavorite test.hlog DeviceTests: 36/36 passed. - KV.benchmark --device randomaccess --log-memory 256m --preallocate-log --rumd 50,50,0,0 --validate: PASS, 357 K ops/sec, iostat shows 100-177 K real disk r/s and 53-90% NVMe util → O_DIRECT page-cache bypass confirmed. - KV.benchmark --device randomaccess (log fits) --validate: PASS, 15.7 M ops/sec. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite tests: cross-device hardening suite (Native + RandomAccess + ManagedLocal) The Phase-7 hardening suite added in this branch was NativeStorageDevice- specific. Add parametrized variants of the four tests that are pure IDevice contract checks (not native-specific lifecycle/API), so they exercise all three local-storage device implementations: - Hardening_AllDevices_RoundTrip_BasicReadWrite - Hardening_AllDevices_RoundTrip_AcrossSegmentBoundary - Hardening_AllDevices_Parallel_32ConcurrentWrites - Hardening_AllDevices_Parallel_BurstyTraffic Each is parametrized by a new DeviceKind enum (Native, RandomAccess, ManagedLocal). Native is gated on OperatingSystem.IsLinux() (the C++ shim links against libaio/liburing); the other two run on both Linux and Windows. A shared CreateDeviceForTest helper takes care of the per-kind ctor + Initialize() dance so the test body stays uniform. Result: 38/38 hardening tests pass on Linux (12 new cross-device + 26 native-only). Native-specific tests retained as-is because they test API that doesn't exist on the other devices: - Lifecycle (DisposeBeforeInitialize, InitializeTwice, etc.) — NativeStorageDevice defers Initialize from the ctor; the other devices initialize in their ctor. - Segment-size validation (NonPowerOfTwoSegmentSize_Throws, etc.) — Initialize() is the only callsite that validates. - Recovery_*_SegmentSize_* — the native device's open() path is the only place that records and re-validates per-segment-size metadata. - SectorSize stability across opens — not all devices expose this. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite device tests: split into IDevice_ contract + NativeStorageDevice_ buckets; fix AsyncPool creator-throw hang Test refactor: - IDevice_*: 8 contract tests parametrized across Native, RandomAccess, ManagedLocal (round-trip basic, round-trip cross-segment, round-trip various segment sizes, 32 concurrent writes, 64 concurrent reads, mixed reads+writes, bursty traffic, stress burst of 100 writes, permission-denied callback contract). 33 cases total. - NativeStorageDevice_*: 16 native-only tests for behaviors managed devices don't have (deferred Initialize signature, recovery segment-size mismatch detection, sector-size discovery, sync-throw unaligned IO guard). AsyncPool fix: GetOrAdd reserved a slot in totalAllocated before calling creator(). If creator() threw (e.g. open() returned EACCES, ENOSPC), the slot was never released, so Dispose() would loop forever waiting for totalAllocated to drain to zero. This manifested as a process hang when a device pool's first open() failed. Rollback the reservation on exception so the failure propagates cleanly and the pool can still be disposed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite IDevice: unify Initialize contract — required for all devices, -1 = unbounded single segment Before this change, IDevice.Initialize had the same signature for every device but very different semantics: * StorageDeviceBase (RandomAccess, ManagedLocal, LocalStorage, NullDevice, LocalMemoryDevice): the ctor pre-set segmentSize = -1 / bits = 64 / mask = ~0, so calling an IO entry point without Initialize() silently ran in unbounded single-segment mode. * NativeStorageDevice: Initialize was MANDATORY (the C++ shim needs the segment size at create time for libaio/io_uring geometry), IO entry points threw if invoked first, and segmentSize = -1 was rejected. This commit unifies the contract: every IDevice must call Initialize() exactly once before any IO, and segmentSize = -1 selects unbounded single-segment mode on every device. Implementation: * StorageDeviceBase - Added `initialized` flag (volatile) and `EnsureInitialized()` helper that throws InvalidOperationException with a clear message naming the device by FileName. - Ctor leaves `initialized = false` but keeps the safe fallback defaults (-1 / 64 / ~0) so any cold maintenance path that touches segmentSizeBits before the guard can't compute outright nonsense. - EnsureInitialized() called from the base address-based ReadAsync / WriteAsync overloads and TruncateUntilAddress / TruncateUntilAddressAsync. - Initialize sets `initialized = true` at the end. * NativeStorageDevice - Accepts segmentSize = -1: translates to 1UL << 63 for the native shim so the C++ FileSystemSegmentedFile's shift = log2(segment_size) math collapses every non-negative upper-layer address into segment 0 (parity with the managed-side bits = 64 / mask = ~0). Single growing file on disk. - Tracks the value passed to native in nativeSegmentSizeBytes (replaces the diagnostic-only configuredSegmentSizeBytes long field, which couldn't hold 1<<63 without overflow). - ABI readback (NativeDevice_GetSegmentSize) compared against the value we sent to native, not the user-facing -1. - Always rejects omitSegmentIdFromFilename — the C++ shim has no omit-suffix code path, every segment is written as <base>.<segmentId>. Better to fail fast than silently produce wrong file names. * Concrete IO entry points (ReadAsync / WriteAsync / RemoveSegment / RemoveSegmentAsync) of NullDevice, LocalMemoryDevice, ManagedLocalStorageDevice, RandomAccessLocalStorageDevice, LocalStorageDevice, AzureStorageDevice, ShardedStorageDevice, and TieredStorageDevice now call EnsureInitialized() before doing work. Caller fix-ups: * LocalStorageNamedDeviceFactory.Get now calls device.Initialize(-1L) before returning. Commit / checkpoint metadata is single growing-file usage (segment 0 only, .0 suffix), so unbounded mode is the right default and unblocks every DeviceLogCommitCheckpointManager caller from needing to remember to initialize. * LocalStorageNamedDeviceFactory.ListContents skips dotfile entries — defensive against a pre-existing race in LinuxFileExtensions.IsDirectIOSupported where a .tsavorite-odirect-probe-* temp file can leak in the commit dir if File.Delete races with File.GetFiles. Without this filter, leaked probe files surface as Int64.Parse("") failures in DefaultCheckpointNamingScheme.CommitNumber. * SimulatedFlakyDevice.Initialize now propagates to the wrapped device. * ComponentRecoveryTests Setup_* helpers call Initialize(-1) on devices they construct directly (bypass the Tsavorite allocator path which normally Initializes). Tests (DeviceTests.cs): * IDevice_ReadAsyncBeforeInitialize_Throws(kind) × 3 — new contract test. * IDevice_WriteAsyncBeforeInitialize_Throws(kind) × 3 — same. * IDevice_Initialize_SegmentSizeMinusOne_UnboundedSingleSegment(kind) × 3 — write at offset 1 MiB (would be in segment-N for any positive size) and read back, confirming -1 routes through segment 0 on all 3 kinds. * NativeStorageDevice_Initialize_OmitSegmentIdFromFilename_Throws — new native-only test for the omit rejection in both -1 and explicit-size modes. * Removed NativeStorageDevice_{Read,Write}AsyncBeforeInitialize_Throws (now subsumed by the IDevice_ variants). Docs: * IDevice.Initialize docstring rewritten to spell out the new contract and the -1 semantics. NativeStorageDevice.Initialize remarks updated. Verified on Linux net10.0 Release: * 599 hlog tests (491 passed + 108 skipped) * 305 recovery tests * 144 + 155 + 127 + 346 = 772 other Tsavorite + Garnet RespTests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite: native devices honor omitSegmentIdFromFilename; O_TMPFILE probe (no race) Two related fixes on top of the unified Initialize contract: 1) NativeStorageDevice now supports omitSegmentIdFromFilename ───────────────────────────────────────────────────────── Previously Native rejected the omit flag because the C++ shim hard-coded the '.<segmentId>' suffix in three places in file_system_disk.h. This made the IDevice contract asymmetric (managed devices honored omit, Native didn't). Fix by threading the bool through the entire C++/C ABI: C++ (libs/storage/Tsavorite/cc/src/device/): * FileSystemSegmentBundle: new bool omit_segment_id_; both ctors accept it and use a new segment_path(idx) helper that returns just filename_ when set, otherwise filename_ + '.' + std::to_string(idx). Used at all three locations that previously hard-coded the suffix. * FileSystemSegmentedFile: new bool omit_segment_id_ (const) wired through ctor and propagated to bundles allocated by OpenSegment. * NativeDeviceImpl: new bool omit_segment_id constructor param; recorded as omit_segment_id_ member. ValidateRecoveredSegments short-circuits in omit mode (single bare-named file, segment-size mismatch check is meaningless when there's no .<id> suffix to scan for). * native_device_wrapper.cc / NativeDevice_CreateWithBackend: new trailing 'bool omit_segment_id' parameter. ABI BUMP — managed wrapper updated to match; Linux .so rebuilt and committed at libs/storage/Tsavorite/cs/src/core/Device/runtimes/linux-x64/native/ libnative_device.so. **Windows DLL must be rebuilt by user** with cmake -G 'Visual Studio 17 2022' -A x64 -T v143,spectre=true. C# (libs/storage/Tsavorite/cs/src/core/Device/): * NativeStorageDevice P/Invoke signature updated. * NativeStorageDevice.Initialize removes the 'always rejects omit' guard. It now accepts omit:true together with segmentSize = -1 and forwards to native; rejects omit:true together with a positive segmentSize with a clear error message (multiple segments would collapse onto the same on-disk path and clobber each other). Tests (DeviceTests.cs): * IDevice_Initialize_OmitSegmentIdFromFilename_BareFileName(kind) × 3: writes via Initialize(-1, omit:true) and asserts the on-disk file is the bare basename (no .0 suffix). Replaces the native-only 'throws' test from the previous commit. * IDevice_Initialize_OmitSegmentIdFromFilename_WithoutMinusOne_Throws(kind) × 3: enforces the no-positive-size-with-omit invariant on every kind. 2) IsDirectIOSupported uses O_TMPFILE (race-free probe) ───────────────────────────────────────────────────── The previous probe in libs/storage/Tsavorite/cs/src/core/Device/ LinuxFileExtensions.cs created a hidden '.tsavorite-odirect-probe-<pid>- <guid>' file in the device's directory, then File.Delete'd it in a silent-catch finally. Multiple concurrent commits (one device per Get()) ran probes simultaneously; concurrent ListContents calls from CommitRecordBoundedGrowthTest would observe the probe file during its brief lifetime, and DefaultCheckpointNamingScheme.CommitNumber would then throw FormatException on long.Parse(''). 18/20 baseline failure rate. Switching from create+unlink to open(directory, O_TMPFILE | O_RDWR | O_DIRECT) tells the kernel to allocate an anonymous inode in the directory's filesystem with NO directory entry. The probe inode is invisible to readdir/getdents regardless of timing; concurrent ListContents cannot observe it. Freed on close. Linux >= 3.11 + ext4/xfs/tmpfs/btrfs all support it. If O_TMPFILE itself fails (EOPNOTSUPP on some filesystem) we conservatively report 'no O_DIRECT' so the device falls back to the page-cache path — no named-file fallback because that's the bug we're fixing. Reverts the dotfile filter in LocalStorageNamedDeviceFactory.ListContents added by the previous commit; the underlying race is now eliminated at the kernel level so the workaround is unnecessary. 20/20 LogFastCommitTests runs pass after the change (was 2/20 on baseline, 3/20 on the previous unlink-after-open attempt which still raced). Verified on Linux net10.0 Release: * 612 hlog tests (504 passed + 108 skipped) * 305 recovery tests * 62 device tests (IDevice contract + NativeStorageDevice-specific) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * KV.benchmark: add disk-IO thread-scale sweep recipe to cookbook Captures the 100M-key disk-bound thread-scale experiment from the optimize-device branch sweep so it can be reproduced verbatim in the future: - 100M × 100B records (12.8 GB on disk) - 16 MB log so ~0.125 % of dataset is in memory and almost every read is a 4 KB random disk fetch - 8 load threads, run-threads sweep 1,2,4,8,16,32 at 15s each - One row per backend (RandomAccess / native+libaio / native+uring) Added a short note after the table explaining what to compare against (the disk's fio ceiling at 4K-aligned QD=64-per-job), the expected ~2 min wall-clock per device, and the observed per-backend plateau characteristics so the next operator knows what 'good' looks like. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Device.benchmark + KV.benchmark: robustness & flag improvements Device.benchmark fixes (previously reported throughput could be 2x inflated): - Throughput counter now tallies successful completions only. Before, every ReadAsync call was counted as success even when the kernel returned EAGAIN (Status::IOError=4 — flooded libaio io_context ring). Under --throttle-limit 0 with high QD, ~40% of "ops" were errored requests. - Per-error-code histogram printed at end of run; no per-error Console.WriteLine (was emitting millions of serial writes/run, both falsifying numbers and slowing the real path). - DEBUG data validation skipped on errored ops (was reading garbage from the destination buffer on EAGAIN paths and reporting spurious "Data mismatch"). - --throttle-limit help text documents the libaio kernel ring trap (128 slots wide; high QD + no throttle floods it) and recommends --throttle-limit 128 (also the io_uring SQ depth this build uses). - --io-backend flag added (libaio / uring / default) so the existing Linux Native path can be exercised against either backend. Unknown values are rejected with an actionable message at startup instead of silently falling back to default. - --completion-threads is now wired through to the Linux Native ctor (was hardcoded to 1). - --file-size widened to long (was int; --file-size > 2GB threw a parse error). KV.benchmark + Devices.cs: clarified XML/help text for the existing --device-completion-threads / numCompletionThreads parameter to describe the current behavior (multiple drainer threads share one kernel io_context / io_uring per device). No behavior change for KV.benchmark or core Tsavorite Devices.cs API beyond the Device.benchmark surface and clarified docs. * Tsavorite Native: shard io_uring per completion thread; new C ABI Adds N independent kernel io_contexts (libaio) / io_urings (uring) per NativeStorageDevice, with completion threads bound 1:1 to contexts. libaio internally always uses 1 context (sharding empirically gave nothing — kernel mutex efficient at all tested loads), but the sharded ABI surface is kept across both backends so the templated NativeDeviceImpl<HandlerT> doesn't need a fork. C ABI changes (libnative_device.so): - NativeDevice_CreateWithBackend signature bumped: trailing int32 num_io_contexts. - New exports: NativeDevice_QueueRunFor(device, ctx_idx, timeout_secs), NativeDevice_NumIoContexts(device). - Legacy NativeDevice_QueueRun kept; under uring sharding it scans all rings (back-compat for any single-thread drainer). C# (NativeStorageDevice): - Synchronous ABI probe at Initialize() that converts EntryPointNotFoundException into a clear TsavoriteException listing the missing exports and how to rebuild — guards against a stale .so silently hanging Dispose's drain loop. - Probe is intentionally gated by the QueueRun branch so it runs only on backends that actually use the new symbols at runtime: Linux Native (libaio / uring) where QueueRun returns >= 0, NOT on Windows IOCP where the ThreadPoolIoHandler returns -1 by design. This means a stale Windows DLL keeps working unchanged because it never calls the sharded exports. cdecl/x64 ABI silently tolerates the new trailing num_io_contexts arg on NativeDevice_CreateWithBackend. - Completion threads bound 1:1 via QueueRunFor(ctxIdx) (no closure capture bug — ctxIdx is captured per-iteration into a local). - numCompletionThreads is the user-facing knob; native side decides how many contexts to actually create (libaio: always 1; uring: honours the request). Empirical justification (Device.benchmark, NVMe, 4K random reads, batch=4096, throt=512, fio ceiling 749K, NUMA0-pinned): libaio CT=1 (always 1 ctx + 1 drainer): 755K ops/sec (t=32) uring CT=1 (1 ring + 1 drainer): 357K ops/sec ← SpinLock-bound uring CT=1 ring + N drainers (regresses): 357K → 274K ← cq_lock contention uring CT=4 (4 rings + 4 drainers): 745K ops/sec uring CT=8 (8 rings + 8 drainers): 758K ops/sec ← hardware ceiling Both backends now reach the hardware NVMe ceiling. uring requires sharding (the user-space SpinLock around io_uring_get_sqe + prep + submit is the real cap). libaio doesn't need sharding (kernel io_context mutex already efficient at all tested loads). Files: - file_linux.h : UringIoHandler sharded (vector<io_uring*>, per-ring sq_lock + cq_lock, atomic round-robin pick_ring). QueueIoHandler unchanged on the data plane (single io_context_t) but exposes the same num_contexts()/TryCompleteFor/QueueRunFor surface as inline stubs for ABI symmetry. - file_linux.cc : new UringIoHandler impls; QueueIoHandler unchanged. - file_windows.h: stub overloads (num_contexts()=1, QueueRunFor=-1, 2-arg ctor) so the templated NativeDeviceImpl compiles unchanged. - native_device.h, native_device_wrapper.cc: ABI plumbing as above. - NativeStorageDevice.cs: ABI probe + per-context drain workers. - runtimes/linux-x64/native/libnative_device.so: rebuilt with sharding. * Device.benchmark: add cookbook README showing how to saturate ~750K NVMe IOPS Captures the verified copy-paste recipe for both Linux Native backends (libaio and io_uring) to hit the hardware ceiling on a Dell P5600-class NVMe, alongside a flag reference, output-schema explanation, and troubleshooting table. Headline recipes verified end-to-end on the reference setup: libaio --completion-threads 1 --threads 16 --throttle-limit 512 → 743K ops/sec uring --completion-threads 8 --threads 16 --throttle-limit 512 → 738K ops/sec (Both within 2 % of the table values in the README; zero kernel-side errors.) Key facts documented: - libaio always uses one io_context in this build regardless of --completion-threads (sharding empirically gave nothing; the hint is ignored). Pass 1 explicitly so scripts are self-describing. - io_uring needs sharded rings (CT >= 4) to escape the per-ring user-space SpinLock cap around io_uring_get_sqe + prep + submit; CT=8 is the safe peak. - --throttle-limit must be set to at least the per-ring/per-context depth (128 in this build for both backends). --throttle-limit 0 floods the kernel ring and the benchmark correctly surfaces Status::IOError=4 in the per-code histogram rather than tallying errored ops as throughput. - --file-size must be a multiple of 1024 × --sector-size (fill phase uses a 1024-sector temp buffer). Plus a section comparing Device.benchmark vs KV.benchmark to direct readers to the right tool: Device.benchmark for IO-layer ceiling validation (saturates NVMe), KV.benchmark for full-path throughput (currently caps ~30 % below the IO ceiling on the upper-layer pending-read path — see KV.benchmark README for that side of the story). Also adds a one-paragraph pointer in benchmark/README.md so the new README is discoverable from the top-level benchmarks listing. * Tsavorite Native: bounded sched_yield retry on transient kernel-ring full ScheduleOperation in both QueueFile (libaio) and UringFile (io_uring) now retries the kernel-side submission on the transient back-pressure signal (libaio: `io_submit == 0`; uring: `io_uring_get_sqe == nullptr`) up to kMaxSubmitRetries = 8 attempts, each separated by a sched_yield(). Permanent errors (libaio io_submit < 0, uring io_uring_submit < 0) are NEVER retried — they surface immediately as Status::IOError. Motivation: the upper-layer throttle gate in AllocatorBase.AsyncGetFromDisk is a racy test-then-increment (Throttle() reads numPending non-atomically, then ReadAsync does Interlocked.Increment). With N concurrent submitters all passing the gate at numPending == ThrottleLimit, in-flight can spike to ThrottleLimit + N momentarily, exceeding the 128-slot per-context / per-ring kernel ring depth when N > 8 (which is normal for Garnet under heavy disk-bound load). Pre-fix, the kernel rejects the overshoot submissions with EAGAIN, which Tsavorite handled by re-routing through the full AllocatorBase pending-read retry loop (correct but expensive: a full round-trip per IOError). Post-fix, the burst is absorbed locally by a handful of sched_yields and never surfaces to the upper layer. Why sched_yield + bounded retries is the right shape: on a 750K-IOPS NVMe the kernel ring drains a slot every ~1.3 µs and sched_yield is typically 1-10 µs on Linux, so 8 retries (worst-case ~40-80 µs window) is more than enough to absorb the typical 24-slot overshoot from a 32-thread burst. For genuine sustained overload (application submission rate exceeds device IOPS for seconds), the retries exhaust and Status::IOError surfaces — which is the correct signal for the caller to apply back-pressure. Implementation notes: - libaio: simple loop around io_submit. No lock to release; io_submit is kernel-thread-safe per io_context, so concurrent submitters serialise inside the kernel. - uring: must release sq_lock around sched_yield. Holding a SpinLock across a syscall would stall every other submitter on the same ring. Only the get_sqe == nullptr path is retried; if get_sqe succeeded we've already "consumed" an SQE slot in the user-side bookkeeping and re-issuing via get_sqe+prep on retry would corrupt the ring (we'd hold two SQEs for one logical op). For SQPOLL-disabled rings (our setup) io_uring_submit returns 1 in steady state — a non-1 there is an unrecoverable kernel-side error and surfaces immediately. Verified end-to-end (Device.benchmark, NVMe, 4K random reads, batch=4096, NUMA0-pinned, --throttle-limit 120 to match production NativeStorageDevice default): libaio CT=1: t=8 / 16 / 32 → 626K / 631K / 627K ok/sec, 0 err uring CT=8: t=8 / 16 / 32 → 622K / 625K / 626K ok/sec, 0 err At extreme intentional-overload settings (--throttle-limit 4096, t=32) errors still appear — confirming the retry budget correctly exhausts when the application is genuinely outpacing the device: libaio CT=1, t=32, throt=4096: 754K ok, 14.7M code4 err (~5% of submits) uring CT=8, t=32, throt=4096: 745K ok, 0 err (uring still produces 0 err under same overload because 8 rings × 128 = 1024 SQ slots is large enough that even gross over-submission fits within the retry budget per ring.) * Tsavorite Native: PR review fixes (3-model code review pass) BLOCKER fixes: - NativeStorageDevice.Dispose UAF race. Previously NativeDevice_Destroy(nativeDevice) ran before nativeDevice was nulled, so a concurrent guard-bypassed P/Invoke could observe a non-zero handle that points to freed memory. Fix: Interlocked.Exchange atomically captures-and-nulls the handle; destroy runs on the captured pointer. EnsureReadyOrSilent now checks disposedFlag first. HIGH fixes: - UringFile::ScheduleOperation SQE leak on transient io_uring_submit failure. After a successful get_sqe + prep, the SQE is committed to the user-side SQ ring; a -EAGAIN/-EBUSY return from io_uring_submit left the slot permanently occupied with no kernel iocb, eventually starving get_sqe forever. Fix: retry io_uring_submit (without re-preparing) up to kMaxSubmitRetries on transient negatives, with sched_yield (and sq_lock released) between attempts. - UringIoHandler::Init partial-init leak. If new SpinLock() threw after io_uring_queue_init succeeded for ring i, the already-initialized ring leaked (the class dtor doesn't run on partial construction). Fix: use std::unique_ptr RAII holders during construction; release into the member vectors only after all allocations succeed. POLISH fixes: - NativeStorageDevice.Initialize tail-throw cleanup. If base.Initialize threw after the native device and completion threads were created, both leaked. Now wrapped in try/catch that cancels token, joins threads, and destroys the native device. - UringIoHandler rule-of-5 hygiene: explicitly deleted copy ctor, copy-assign, and move-assign so the implicit shallow copies (which would double-delete the raw owning pointers) cannot be generated. - DispatchUringCqe: added static_assert(is_trivially_destructible<IoCallbackContext>) so a future non-trivial member fails the build instead of silently leaking. - NativeDeviceImpl::num_io_contexts: removed unnecessary const_cast (the underlying num_contexts() is already const on both backends). - Removed duplicate XML <summary> on NativeStorageDevice.Dispose. - FileSystemDisk dead-code ctor: passed the now-required 5th arg to FileSystemSegmentedFile so the file compiles if anyone instantiates it. Comment hygiene sweep (per explicit review-rule #4 "comments should not refer to design thought processes"): - Removed embedded benchmark results, hardware-specific throughput numbers, and historical narrative from class/method documentation in file_linux.{h,cc}, native_device.h, native_device_wrapper.cc, NativeStorageDevice.cs. - Kept WHAT each method does and the invariants it enforces; moved WHY this approach was chosen out of source comments (the commit log is the appropriate place for that context). - Net: -162 lines across 7 files, no behavior change from the trim itself. Verified post-fix performance unchanged (Device.benchmark, NVMe, 4K random reads, batch=4096, NUMA0-pinned): libaio CT=1 t=16 throt=512: 746K ok/sec, 0 err (hardware ceiling) uring CT=8 t=16 throt=512: 743K ok/sec, 0 err (hardware ceiling) libaio CT=1 t=32 throt=120: 636K ok/sec, 0 err (production default) uring CT=8 t=32 throt=120: 618K ok/sec, 0 err (production default) * Tsavorite Native: make Initialize idempotent via lazy native-handle creation NativeStorageDevice.Initialize used to perform all the heavy work (native device creation, completion-thread spawn, ABI / segment-size / sector-size cross-checks) eagerly and threw "called more than once" on a second call. The other IDevice implementations (LocalStorageDevice, RandomAccessLocalStorageDevice, ManagedLocalStorageDevice) all inherit a metadata-only StorageDeviceBase.Initialize that simply overwrites segmentSize / segmentSizeBits / mask fields and is silently idempotent. They open their per-segment OS handles lazily inside the IO methods. This contract mismatch broke any caller that invokes Initialize twice on the same NSD instance. The canonical case is LocalStorageNamedDeviceFactory.Get(), which calls Initialize(-1L) as a defensive pre-init so consumers can't forget; the consumer (snapshot checkpoint state machine SnapshotCheckpointSMTask, cluster checkpoint streaming TsavoriteCheckpointReader.CreateCheckpointDevice) then calls Initialize(actualSegmentSize). Under the old NSD that throws; under the new NSD it works the same way the other backends do. Implementation: - NSD.Initialize is now metadata-only — delegates to base.Initialize. Pre-flight argument validation (segmentSize power-of-two, sector-size floor, omitSegmentIdFromFilename) is preserved. - New EnsureNativeDeviceCreated() does the heavy work, lazily, on first IO. Reads the latest base.segmentSize and base.OmitSegmentIdFromFileName so whichever Initialize call ran most recently wins. - Thread-safe via double-checked locking on a new nativeCreateLock. The publish of nativeDevice uses Volatile.Write so a second observer of nativeDevice != IntPtr.Zero is guaranteed to see a fully-initialised handle with completion threads already running. - Dispose now also takes nativeCreateLock around the cancel-join-destroy sequence so it cannot race with a concurrent EnsureNativeDeviceCreated (which would otherwise leak a freshly-published native handle and its completion threads). - IO entry points (ReadAsync, WriteAsync) call EnsureNativeDeviceCreated() before submission. Bookkeeping entry points (Reset, TryComplete, GetFileSize, RemoveSegment) no-op when the native handle has not been created yet, matching the semantics of the other backends (Reset on a device with no open handles is a no-op). Verified against the full unit-test sweep with Native forced as the default device (the GetDefaultDeviceType hack is local-only and not in this commit): Tsavorite.test: 206 / 206 (was 204 / 206 pre-fix) Garnet.test: 789 / 789 (was 110 / 792 pre-fix; 681 were blocked on Initialize-twice) Garnet.test.acl: 425 / 425 Garnet.test.collections: 746 / 746 Garnet.test.complexstring: 386 / 386 Garnet.test.rangeindex: 62 / 62 Garnet.test.vectorset: 42 / 42 No change to behavior for callers that invoke Initialize once with the real segment size, which is what every production code path already does. * Tsavorite Native: probe GetFileSize/RemoveSegment without forcing native handle GetFileSize and RemoveSegment must report the on-disk state regardless of whether IO has flowed through the device, matching LocalStorageDevice and RandomAccessLocalStorageDevice semantics. Before this fix, both no-op'd when no native handle had been created — which silently truncated the cluster manager's recovery decision because ClusterManager.cs:79 and ReplicationManager.cs:160 call `device.GetFileSize(0) > 0` to decide whether to recover persisted cluster config / replication history. With Native, a restarted node would always "Initialize new node instance config" instead of recovering, get a fresh node ID, and fail every replication-resume test (e.g. ClusterSRNoCheckpointRestartSecondary which restarts a replica and then waits for AOF sync to catch up). Changes: * GetFileSize now falls back to FileInfo when no native handle exists — same shape as RandomAccessLocalStorageDevice.GetFileSize (open-on-demand) but without paying io_uring/libaio setup cost just to stat a file. * RemoveSegment now falls back to File.Delete when no native handle exists — same shape as LocalStorageDevice / RandomAccessLocalStorageDevice (best-effort unlink, swallows ENOENT). * Per IDevice contract enforced in 889def4 ("Tsavorite IDevice: unify Initialize contract — required for all devices, -1 = unbounded single segment"), ReadAsync / WriteAsync now call EnsureInitialized() before EnsureNativeDeviceCreated() so the IDevice_*BeforeInitialize_Throws hardening tests get the same InvalidOperationException shape from Native that they get from the other devices. * Two device tests updated to match the lazy-Initialize contract that was introduced in commit f4e3044 ("Tsavorite Native: make Initialize idempotent via lazy native-handle creation"): - NativeStorageDevice_InitializeTwice_Throws → _Idempotent: idempotent Initialize matches the LSD/RA contract used by LocalStorageNamedDeviceFactory.Get + consumer re-init pattern. - NativeStorageDevice_Recovery_LargerExistingSegment_DetectsMismatch: the C++ ValidateRecoveredSegments check now fires on first IO (when EnsureNativeDeviceCreated runs), not at Initialize time, so the test asserts on a ReadAsync rather than Initialize. Verified on Linux x64 / .NET 10: * libs/storage/Tsavorite/cs/test/test.hlog: all IDevice + NativeStorageDevice tests pass (62/62) with Native default * test/cluster/Garnet.test.cluster.replication: all 4 ClusterSRNoCheckpointRestartSecondary variants pass with Native default (regression-test for the recovery path) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: wake completion drainer on Dispose via no-op IO Without this fix, NSD.Dispose() can stall up to CompletionWorkerTimeoutSecs (1s) per io_context because the completion-drainer thread is blocked in io_getevents / io_uring_wait_cqe_timeout waiting for events that will never come (the IO drain phase has already brought numPending to 0). The Thread.Join following completionThreadToken.Cancel() then has to wait for the next QueueRunFor timeout to fire so the thread can observe cancellation and exit. This was visible as exactly-1.0s gaps in cluster replication recovery traces: checkpoint metadata reads / writes that each create+dispose a fresh NSD spent ~1s in Dispose, multiplying across the ~5–10 devices created per checkpoint into multi-second stalls. ClusterReplicaSyncTimeoutTest (replicaSyncTimeout=1s) and MultiDatabaseSaveRecoverByDbIdTest(True) (2s LASTSAVE poll window) failed because of this; the actual I/O on Native is microseconds, not seconds. Fix: post a synthetic wake-up event on each io_context when Dispose runs. * libaio: submit a 0-byte read on a /dev/null fd opened in the handler ctor. /dev/null completes immediately and does not require O_DIRECT alignment, so the wake-up does not interfere with the real segment files. * io_uring: submit io_uring_prep_nop with user_data = nullptr; the drain loop recognises nullptr as a wake-up sentinel and skips dispatch. * Windows ThreadPoolIoHandler has no dedicated drainer (callbacks fire on threadpool threads), so its Wake is a no-op stub returning 0. The completion thread wakes from its blocking syscall almost immediately, observes the cancellation token on its next loop iteration, and exits. No extra idle work, no polling, no shortened timeout. * NSD.Dispose latency: 1025ms worst case -> ~20-30ms (microbenchmark). * ClusterReplicaSyncTimeoutTest with Native: ~22-25s (fail) -> ~3s (pass). * MultiDatabaseSaveRecoverByDbIdTest(True) with Native: timeout (fail) -> ~6s (pass). * Idle drainer syscall rate is unchanged (1/s/context). C ABI changes (additive — old exports preserved): * NativeDevice_WakeCompletionWorker(device, ctx_idx). * INativeDevice::Wake; QueueIoHandler::Wake, UringIoHandler::Wake, ThreadPoolIoHandler::Wake stub. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite IDevice: make Initialize() optional — ctor defaults are valid for IO Background: commit 889def4 ("unify Initialize contract — required for all devices") added an EnsureInitialized() guard that threw InvalidOperationException at every IO entry point if Initialize() had not been called first. This was redundant: the ctor already establishes segmentSize=-1 / segmentSizeBits=64 / segmentSizeMask=~0UL, which is functionally identical to having called Initialize(-1) — every absolute address right-shifts to segment 0, producing unbounded single-segment routing. The mandatory-Initialize contract was the root cause of the entire factory-pre-init + NSD lazy-creation saga: LocalStorageNamedDeviceFactory.Get was forced to call device.Initialize(-1L) defensively just to satisfy the contract, which broke NativeStorageDevice (its single-shot Initialize then asserted on the consumer's follow-up Initialize(realSize)). Recent commits f4e3044 + 0535da0 papered over this with lazy native-handle creation; this commit removes the root cause. Changes: * StorageDeviceBase: remove the 'initialized' flag, EnsureInitialized() helper, and ThrowNotInitialized() method. Initialize() is now purely a *configuration* call to override the ctor defaults (set a non-default segment size, opt into OmitSegmentIdFromFileName). The ctor doc explicitly states that callers may issue IO immediately after construction. * All concrete devices: remove the EnsureInitialized() calls at the top of ReadAsync / WriteAsync / TruncateUntilSegmentAsync / RemoveSegment (libaio, io_uring, RA, ManagedLocal, LocalMemory, Null, Tiered, Sharded, Azure). * LocalStorageNamedDeviceFactory.Get: drop the defensive device.Initialize(-1L); the ctor defaults match what that call did anyway. * NSD's recent EnsureInitialized() additions to ReadAsync/WriteAsync (introduced in 0535da0 only to satisfy the hardening test) are also removed by the sweep. * ComponentRecoveryTests.cs: drop 3 redundant Initialize(-1L) calls. * test.hlog DeviceTests: rename and repurpose IDevice_*BeforeInitialize_Throws to IDevice_*BeforeInitialize_UsesCtorDefaults — the new test demonstrates that WriteAsync/ReadAsync on a freshly-constructed device (no Initialize call) works correctly using the unbounded single-segment defaults, across Native / RA / ManagedLocal. TestUtils.cs:165 still calls device.Initialize() — that path is conditional on the caller wanting OmitSegmentIdFromFileName=true, which IS only settable via Initialize (it is not a ctor parameter), so the call is genuinely needed there. Verified on Linux x64 / .NET 10: * libs/storage/Tsavorite/cs/test/test.hlog (IDevice + NativeStorageDevice tests): 62/62 pass. * libs/storage/Tsavorite/cs/test/test.recovery (ComponentRecovery tests): 4/4 pass. * Full Garnet.test, Garnet.test.cluster, Garnet.test.acl, Garnet.test.collections, Garnet.test.extensions, Garnet.test.scripting, Garnet.test.complexstring, Tsavorite IDevice+NSD: pass at the same rates as before this commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native + IDevice: refresh comments to reflect final design Sweep the PR for comments that referenced earlier design choices as they evolved during development, and rewrite them to describe the steady-state contract directly without historical baggage. * IDevice.Initialize: replace the "Must be called exactly once... before any IO entry point... uninitialized device throws InvalidOperationException" doc with the actual contract: Initialize is purely an opt-in configuration step to override the ctor defaults (which are equivalent to Initialize(-1)); callers may issue IO immediately after construction. * NativeStorageDevice ctor doc: rewrite to describe the as-shipped lazy creation flow (configuration captured at ctor, native handle created on first IO via EnsureNativeDeviceCreated) rather than the stale "Native device creation is DEFERRED until Initialize... every IO entry point throws InvalidOperationException" framing. * NativeStorageDevice.Initialize doc: drop the misleading "Creates the underlying native device with the requested segment size" lead-in (which hasn't been true since the lazy-creation refactor); replace the "factory pre-init" example (factory no longer pre-inits) with a steady-state description of when repeat Initialize calls are honoured. * NativeStorageDevice.EnsureNativeDeviceCreated doc: replace "Throws if Initialize has not been called" (now uses ctor defaults if no Initialize) with "Throws if the device has been disposed or if the native shim rejects the configuration". * NativeStorageDevice.EnsureReadyOrSilent doc: drop the "does not throw on 'not initialized yet'" qualification. * NativeStorageDevice.GetSectorSize doc: re-point the "cross-check" link from Initialize to EnsureNativeDeviceCreated (which is where it actually happens). * NativeStorageDevice.Dispose doc + body: bound the worst-case shutdown stall by the longest in-flight user callback (not CompletionWorkerTimeoutSecs) since wake-up uses NativeDevice_WakeCompletionWorker; rewrite the inline Dispose comment so it documents the steady-state design rather than what it improved over. * NativeStorageDevice nativeSegmentSizeBytes / UnboundedNativeSegmentSizeBytes field doc: clarify that the value is populated by EnsureNativeDeviceCreated (not Initialize) and that the default is reached without calling Initialize. * NativeStorageDevice_InitializeTwice_Idempotent test: drop the now-stale "factory pre-init... consumer re-initializes" rationale; describe the idempotent contract directly. * NativeStorageDevice_DisposeBeforeInitialize_IsNoOp test: drop the "Phase 6" reference and reword in terms of the steady-state lazy-creation contract. * SimulatedFlakyDevice.Initialize: replace the "so its EnsureInitialized() guard passes when our IO methods delegate to it" comment (the guard no longer exists) with a description of why both devices need matching geometry. * LinuxFileExtensions.OpenDirect dsync param doc: drop the "previously asked for it" wording — the WriteThrough callsites still pass it; describe the parameter as an opt-in for WriteThrough-equivalent semantics. * Doc-cref bookkeeping: change <see cref="base.segmentSize"/> (illegal cref for inherited fields) to <c>base.segmentSize</c> code spans. No behaviour change. Build clean on Garnet.slnx and Tsavorite.slnx; dotnet format --verify-no-changes clean on both. IDevice + NSD device tests all pass (62/62). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: ship libnative_device.so without liburing dependency Background: the shipped libnative_device.so was built with -DUSE_URING=ON, so it had a hard DT_NEEDED entry for liburing.so.2. Loading the .so on a host without liburing2 installed (e.g. a GitHub Actions ubuntu-latest runner, or an end-user box where only libaio is in the base image) failed with: System.DllNotFoundException: ... liburing.so.2: cannot open shared object file: No such file or directory …even for callers that only ever requested the libaio backend, because the dynamic linker resolves NEEDED libraries at load time regardless of which exported symbols the caller goes on to invoke. Rebuild the prebuilt with -DUSE_URING=OFF so the shipped .so links only libaio. Most Linux distributions ship libaio in the base system, so the prebuilt now loads without any additional setup. The io_uring backend becomes a build-time opt-in: callers that want it install liburing-dev and rebuild with -DUSE_URING=ON. The C# layer already surfaces a clear TsavoriteException for callers that request Uring against a USE_URING=OFF build ("Requested IO backend 'Uring' is not available in the loaded native_device library… Rebuild the native library with -DUSE_URING=ON and install liburing-dev to enable io_uring."). Build fix: file_linux.h now includes <fcntl.h> directly (for the ::open() / O_RDONLY usage in QueueIoHandler::OpenWakeFd()). Previously these were pulled in transitively through <liburing.h>, which is now gated behind #ifdef FASTER_URING. Dockerfile updates: drop liburing2 / liburing from the runtime install list in all 5 Dockerfiles (default, .ubuntu, .alpine, .azurelinux, .chiseled). Comments left for users that rebuild with USE_URING=ON. README updates: rewrite the "Runtime dependencies" section to describe the new default (libaio only). Replace the "Disabling io_uring (optional)" section with "Enabling io_uring (optional)". Verified on Linux x64 / .NET 10: libaio default works (62/62 IDevice + NativeStorageDevice tests pass); ldd confirms only libaio.so.1t64 is in NEEDED. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: ship two .so flavors — uring-enabled + libaio-only fallback Single-shipping libnative_device.so created a deployment dilemma: build with USE_URING=ON and end-users without liburing get DllNotFoundException at load time; build with USE_URING=OFF and the io_uring backend stops working even on hosts that DO have liburing installed (which is the case where uring matters for perf — modern NVMe at >1M IOPS benefits noticeably from uring over libaio). Ship both flavors instead: * libnative_device.so — built with USE_URING=ON; DT_NEEDED on libaio AND liburing. Exposes both Libaio and Uring backends. * libnative_device_libaio.so — built with USE_URING=OFF; DT_NEEDED on libaio only. Exposes the Libaio backend. NativeStorageDevice's DllImportResolver tries the uring-enabled binary first; on DllNotFoundException matching 'liburing.so.2: cannot open' it falls back to the libaio-only binary. The Libaio backend therefore always works out of the box on any Linux distribution that ships libaio (essentially all of them). liburing is opt-in: hosts that install it get the Uring backend with zero runtime overhead vs Libaio (direct calls, no function-pointer indirection — we deliberately rejected the dlopen approach so the future-default uring path stays optimal). If a caller explicitly selects IoBackend.Uring on a host without liburing, the construction-time error message now points at the install command per distro ('apt-get install -y liburing2', 'dnf install -y liburing', 'apk add liburing') instead of telling the user to rebuild the .so with -DUSE_URING=ON. We never silently downgrade Uring to Libaio. Changes: * NativeStorageDevice.cs: new LibaioFallbackLibraryPath; ImportResolver catches DllNotFoundException for liburing.so.2 and falls back to the libaio-only .so. ResolveNativeLibraryPath now takes the path as a parameter so it can resolve either flavor. * NativeStorageDevice.cs: rewrite the 'backend not available' exception message — point at install commands (the actual remediation) not rebuild. * Tsavorite.core.csproj: add libnative_device_libaio.so as a second ContentWithTargetPath asset so both .so files are copied to the output directory and packed into the NuGet runtime payload. * runtimes/linux-x64/native/libnative_device.so — REPLACED with USE_URING=ON build (DT_NEEDED libaio + liburing). 2.3 MB. * runtimes/linux-x64/native/libnative_device_libaio.so — NEW, USE_URING=OFF build (DT_NEEDED libaio only). 1.6 MB. * Dockerfile, Dockerfile.ubuntu, Dockerfile.alpine, Dockerfile.azurelinux, Dockerfile.chiseled: re-add liburing2 / liburing to the runtime installs so docker users get the io_uring backend out of the box (the libaio-only fallback would otherwise leave Uring unusable inside containers). * cc/README.md: rewrite the 'Runtime dependencies' and build sections to describe the two-flavor layout, drop the stale 'Enabling io_uring' section, and document the prebuilt rebuild workflow. Verified end-to-end: * Both backends saturate the Dell P5600 NVMe at ~743K random read IOPS in benchmark/Device.benchmark (matches the pre-change reference). * 62/62 IDevice + NativeStorageDevice tests pass. * dotnet format clean on both Garnet.slnx and Tsavorite.slnx. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native (Windows): add init_errno()/initialized() stubs to ThreadPoolIoHandler NativeDeviceImpl's constructor (native_device.h:100-101) gates on handler_.init_errno() to surface an actionable error message when the underlying IO handler failed to initialize (e.g., libaio io_setup() failed with EMFILE / ENOMEM, or io_uring_queue_init() failed). The Linux handlers QueueIoHandler and UringIoHandler both expose this API; ThreadPoolIoHandler (Windows) did not, so MSVC failed to instantiate NativeDeviceImpl<ThreadPoolIoHandler> with: error C2039: 'init_errno': is not a member of 'FASTER::environment::ThreadPoolIoHandler' Add init_errno() and initialized() stubs that return 0 / true unconditionally — the Windows ThreadPool API does not have a separable init step that can fail in the same way the Linux io_setup / io_uring_queue_init paths can (threadpool creation failures propagate via threadpool_'s ctor, not via a later 'check this' field on the handler), so the stubs are semantically correct. NativeDeviceImpl then falls through to the log_.Open(&handler_) path which is where Windows-specific errors (missing directory, permission denied, etc.) actually surface. Linux unaffected: rebuilt build/Release-uring cleanly after this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native tests: stop skipping NSD tests on Windows The Native tests in DeviceTests.cs had blanket 'NativeStorageDevice is Linux-only' Assert.Ignore guards that dated from when NSD's C++ shim was Linux-only. The shim is built on Windows too (native_device.dll via the ThreadPool / IOCP backend in file_windows.cc), so directly constructing 'new NativeStorageDevice(...)' works on Windows. The blanket guards were silently dropping ~15 NSD test cases on Windows CI. Drop the guards so the tests exercise the Windows C++ shim. The legitimate Linux-only guard on IDevice_PermissionDeniedAtFirstWrite_CallbackGetsError (chmod-based; chmod has no Windows analogue) is preserved. Important: end-user device routing is UNCHANGED. Devices.CreateLogDevice(DeviceType.Native) on Windows still returns LocalStorageDevice (managed Windows IOCP), not NativeStorageDevice — that routing happens in Devices.cs and was not touched. These tests directly instantiate the NSD class for shim-coverage purposes only; they do not affect what end users get from the default device factory. Linux: 62/62 pass after this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: rebuild win-x64 native_device.dll for the latest C++ source Rebuild the shipped Windows prebuilt from the current native_device source so the latest fixes (Initialize idempotence, WakeCompletionWorker, etc.) are reflected in the win-x64 DLL. USE_URING is a no-op on Windows; the DLL only exposes the Default (IOCP) backend, so there is no equivalent of the libnative_device_libaio.so fallback on this platform. End-user device routing is unchanged: Devices.CreateLogDevice(DeviceType.Native) on Windows still returns LocalStorageDevice (managed Windows IOCP). This DLL is exercised by direct 'new NativeStorageDevice(...)' construction (see Tsavorite.test.hlog DeviceTests — 59/59 Native + IDevice tests pass on Windows after this rebuild). Built with: Visual Studio 17 2022, MSVC v143, x64, Release configuration, Spectre-mitigated CRT. * Tsavorite Native: address GPT-5.5 PR review findings Fixes two correctness issues caught by an automated code review of the optimize-device PR. ### io_uring SQE leak on submit failure In UringFile::ScheduleOperation, io_uring_get_sqe() advances the user-side sqe_tail before io_uring_submit() is called. If submit fails after retries (-EAGAIN/-EBUSY exhausted, or any other negative), the old code released the lock and returned IOError without doing anything about the still-pending SQE. user_data on that SQE pointed at the io_context unique_ptr that was about to be freed by the guards unwinding, so the next successful submit on the same ring would consume the stale SQE and the QueueRunFor drain loop would dispatch a callback against freed memory — a clear use-after-free. Fix: before releasing sq_lock on the failure path, rewrite the still- pending SQE in place as io_uring_prep_nop with user_data = nullptr. The drain loop already skips nullptr user_data (it's the wake-up sentinel used by UringIoHandler::Wake), so when a later submit flushes this nop the CQE is drained harmlessly. Safe to mutate the SQE in place because we still hold sq_lock and no kernel/concurrent submitter has observed it yet. ### NativeDevice sector_size always returned 512 FileSystemSegmentedFile::alignment() returned a hard-coded 512. NativeDeviceImpl::sector_size() delegated to it, so the C# wrapper's sector-size cross-check in EnsureNativeDeviceCreated would: - falsely throw on 4K-native disks where ProbeAlignment returns 4096 (managed 4096 vs native 512 → 'sector-size mismatch' → device unusable), or - on 4K-native disks where the managed probe fell back to 512 (e.g. older kernel without STATX_DIOALIGN), let the device initialize with SectorSize=512 and then have the kernel reject the 512-aligned O_DIRECT buffers with EINVAL. Fix: factor the STATX_DIOALIGN probe from NativeDevice_ProbeAlignment into a shared inline helper (native_device::ProbeDioAlignment in native_device.h) and call it once from the NativeDeviceImpl ctor, caching the result as the immutable member device_alignment_. sector_size() now returns the cached value; NativeDevice_ProbeAlignment delegates to the same helper. Both sides of the ABI cross-check go through identical probe logic, so the check is now a meaningful ABI / runtime-drift detector instead of a 4K-disk footgun. ### Stale IDevice.Initialize XML The omitSegmentIdFromFilename param said it was 'only supported by managed devices — NativeStorageDevice rejects this flag'. Native devices have honored the flag since 6584cf7. Updated the doc. ### Alpine install hint The 'IoBackend.Uring with libaio fallback' error message suggested 'sudo apk add liburing' on Alpine, but README.md notes that the prebuilt won't load on Alpine (musl) at all. Replaced the apk suggestion with the actual Alpine support story (use a glibc image or fall back to a managed device). Verification: 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass on Linux. Both .so binaries rebuilt (uring-enabled and libaio-only fallback) with correct ldd output. Device.benchmark NVMe saturation throughput unchanged within noise (libaio 738K IOPS, uring 349K IOPS on Dell P5600). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: probe sector size via sysfs max(logical, physical) Replaces the statx(STATX_DIOALIGN) probe in ProbeDioAlignment with a direct sysfs lookup of max(logical_block_size, physical_block_size). Why: - STATX_DIOALIGN reports only the kernel-enforced minimum (= logical block size). It misses the firmware's preferred sector (physical_block_size), so on a 512e drive (logical=512, physical=4096) the probe would return 512 and Tsavorite would take a firmware RMW penalty on every partial-sector write. - STATX_DIOALIGN also requires kernel 6.1+ AND the filesystem to populate the field; ext4 on 6.8 leaves it unset on 512-byte devices, so the probe was already falling through to the 512 default in practice. - sysfs gives us both values directly, on every kernel, with no O_DIRECT dance. Taking max(logical, physical) covers the correctness floor (logical = kernel-enforced minimum) and the performance floor (physical = avoid RMW on partial writes) in one shot. Implementation: - stat() the file (or its closest existing ancestor — log file may not exist yet at construction). Extract st_dev → (major, minor). - Read /sys/dev/block/<maj>:<min>/queue/{logical,physical}_block_size. For partitions (e.g. sda2), the queue/ dir lives on the parent whole-disk block device — fall through to ../queue/<field>. - Round result up to a power of two (always already pow2 on real hardware) and floor at 512 B. On this machine (Dell P5600 NVMe + PERC sda): Probe(/DATA2/badrishc) = 512 (NVMe: logical=512, physical=512) Probe(/tmp/devbench) = 512 (sda partition via parent-walk) Probe(/home/badrishc) = 512 Probe(/) = 512 All values match max(logical, physical) read directly from sysfs. Verification: - 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass - Both .so flavors rebuilt (uring-enabled + libaio-only fallback) - C ABI NativeDevice_ProbeAlignment delegates to the same helper, so managed SectorSize and native sector_size() remain in lockstep. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native tests: align test buffers to 4096, matching sysfs-probed sector size CI failure on IDevice_Initialize_OmitSegmentIdFromFilename_BareFileName (and likely other IDevice_* tests on the same runner) reported: 'NativeStorageDevice.WriteAsync: misaligned I/O — sector size is 4096, but offset=0x0, length=4096, buffer=0x...7EC7A9F76800' The buffer ends at 0x...800 = 2048 — i.e. 2048-aligned but not 4096-aligned. The test helper allocated buffers aligned to HardeningSectorSize = 512 (the pre-PR default for every Garnet Linux device); a 512-aligned formula can land on a 2048-boundary that is not also a 4096-boundary. CI's underlying disk reports physical_block_size = 4096 in sysfs, so the new max(logical, physical) probe returns 4096 there. The native shim then correctly rejects sub-4096-aligned O_DIRECT buffers with EINVAL. The fix is on the test side: bump HardeningSectorSize from 512 to 4096 so the test buffer alignment matches the strictest device.SectorSize seen on any modern hardware (512n, 512e, 4Kn). Locally (Dell P5600 NVMe, logical=physical=512 → SectorSize=512) all 62 IDevice + NativeStorageDevice tests still pass — 4096 trivially divides 512. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite: revert GetAndPopulateReadBuffer changes (defer to separate PR) The read-window sizing optimization (drop leading-slop padding + clamp to page-end) is out of scope for this device-backend PR; it interacts with the larger read-IO path and deserves its own focused PR with dedicated benchmarking. Reverting to the pre-PR behavior here. TryAllocateRetryNow's bounded-backoff change is retained — it's a self-contained allocator hot-path fix and is independently verified (+13.9% on YCSB load with libaio at 64 threads, per kvbench benchmarking). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: Windows probe uses IOCTL_STORAGE_QUERY_PROPERTY for max(logical, physical) Symmetry with the Linux sysfs probe — Windows now reads both BytesPerLogicalSector and BytesPerPhysicalSector from the volume's STORAGE_ACCESS_ALIGNMENT_DESCRIPTOR (via IOCTL_STORAGE_QUERY_PROPERTY + StorageAccessAlignmentProperty) and returns the rounded-up-to-pow2 max, floor 512 B. Previously the Windows branch returned 512 unconditionally, which would silently undersize SectorSize on Windows 4Kn / 512e drives. Implementation: - Parse drive letter from filename ("C:\foo.dat" -> "\\.\\C:"). UNC paths are not supported by this probe — fall back to 512. - CreateFile on the volume with FILE_READ_ATTRIBUTES (no admin needed). - DeviceIoControl(IOCTL_STORAGE_QUERY_PROPERTY) with StorageAccessAlignmentProperty. - max(logical, physical), round up to pow2, floor 512. Linux behavior unchanged. Both .so flavors rebuilt and pass 62/62 device tests on this machine (logical=physical=512 NVMe). REQUIRES Windows DLL rebuild — the Windows path in ProbeDioAlignment is now non-trivial, and the existing prebuilt native_device.dll still returns 512 unconditionally. Without the rebuild, on a Windows 4Kn box the managed SectorSize cross-check would (incorrectly) pass at 512 while the device might actually need 4096. Rebuild recipe in the companion review comment / Tsavorite/cc/README.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add binaries * Tsavorite Native: skip wake-up/failed-submit sentinel CQEs in TryCompleteFor Addresses Copilot review comment on file_linux.cc:373. UringIoHandler::TryCompleteFor (and via it TryComplete) dispatched every drained CQE through DispatchUringCqe without checking the user_data = nullptr sentinel that QueueRunFor already handles. The sentinel marks two kinds of no-op CQEs: - Wake-up nops submitted by UringIoHandler::Wake to unblock the drainer on Dispose. - SQEs rewritten in-place after io_uring_submit failed (the SQE leak fix in c6d68925); these are committed to the SQ but carry no caller context. If a TryComplete() / TryCompleteFor() call picks up either kind of nop CQE, DispatchUringCqe would dereference the null context at context->callback(...) and segfault. Fix: mirror the nullptr-skip from QueueRunFor in TryCompleteFor. Return true to count the drain (matching the any-flag semantics). Verification: 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass. uring .so rebuilt; libaio-only .so is byte-identical because the patched code path is wrapped in #ifdef FASTER_URING and not compiled into the libaio-only fallback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native (uring): batch-drain CQEs and dispatch outside cq_lock QueueRunFor used to acquire cq_lock per CQE (peek -> read fields -> cqe_seen -> release -> dispatch). With a single drainer thread that serializes lock acquire/release on every completion and forces submitters to wait through callback latency when they need ring access. Replaced with the canonical liburing batch-drain idiom: - acquire cq_lock once - io_uring_peek_batch_cqe(ring, cqes, 64) to pull up to 64 CQEs - snapshot (io_res, context) for each - io_uring_cq_advance(ring, n) to release the slots - release cq_lock - dispatch callbacks outside the lock This is the io_uring equivalent of libaio's io_getevents(n) per syscall. Snapshot BEFORE cq_advance is mandatory because the kernel may reuse CQ slots once advanced, leaving the cqe pointers dangling. The wake-up / failed-submit sentinel (user_data == nullptr) is still skipped without dispatch, same as before. Measured impact on Dell P5600 (16 submitter threads, batch 64, throttle 256): ct=1 (1 ring, 1 drainer): 339K -> 354K ops/sec (+4%) ct=4 (4 rings, 4 drainers): 735K -> 737K (saturates, noise) ct=8 (8 rings, 8 drainers): 750K -> 742K avg (saturates, noise) The single-drainer gain is modest because the real bottleneck at ct=1 with 16 submitters is sq_lock contention on the single ring, not cq_lock contention. The batch-drain is still strictly better: - dispatches outside the lock so submitters aren't blocked by user-callback latency, - matches the idiomatic liburing pattern, - amortizes the lock acquire/release across up to 64 CQEs per cycle. For high-throughput workloads, sharding across multiple rings remains the right scaling lever (ct >= 4 saturates this drive). Verification: 62/62 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass. uring .so rebuilt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native (uring): per-thread ring affinity + 4 default rings Eliminates the sq_lock contention that was capping uring at ~340K IOPS at the default numCompletionThreads=1. Two changes work together: 1. Per-thread ring affinity in pick_ring (file_linux.h): Each submitter thread is assigned a ring on its first submit (round- robin against other threads via an atomic counter) and keeps that assignment for life. Same-thread submits never contend on sq_lock with themselves; different threads only contend when they got assigned the same ring (num_submitter_threads > num_rings). This is the user-space equivalent of libaio's "io_submit is thread-safe per io_context" — eliminate shared mutable state across submitters. 2. Hardcoded 4 rings for uring (NativeStorageDevice.cs): numIoContextsConfig = ioBackend == Uring ? max(kDefaultUringRings=4, numCompletionThreads) : numCompletionThreads So uring always has at least 4 rings even at numCompletionThreads=1. The single drainer covers all 4 rings via the legacy QueueRun compat scanner (CompletionWorker passes ctxIdx=-1 in that case). libaio is unchanged: rings == numCompletionThreads (extra rings don't help; the kernel io_context mutex is already efficient). Result on Dell P5600 NVMe (16 submitter threads, batch 64, throttle 256): Before (1 ring, 1 drainer): ~340K After (4 rings, 1 drainer, default): ~700K (matches libaio ct=1) After (8 rings, 8 drainers, sharded): ~745K (unchanged, was already saturating) No new public configuration parameters. numCompletionThreads still controls drainer count; the ring count is now backend-derived behind the scenes. The CompletionWorker single-drainer-multi-ring path was added specifically so the default numCompletionThreads=1 case can saturate without spawning extra drainer threads. Also: bumped HardeningSectorSize and the legacy bufferPool / NativeDeviceTest2 sector_size constants from 512 to 4096 to match the strictest device SectorSize we expect on any modern hardware (4Kn drives where the new max(logical, physical) probe returns 4096). Tests would otherwise fail with EINVAL on 4Kn CI runners with 512-aligned buffers. Verification: - 64/64 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass - Default uring (no flags) hits 626-740K across t=1..64 vs ~340K before - Sharded ct=4/8 unchanged (still saturates) - libaio default unchanged Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add binaries * Tsavorite Native tests: fix ReadInto length mismatch surfaced by 4096 SectorSize NativeDeviceTest1 read 1024 bytes (entryLength) using ReadInto, which: - rounded the read length up to the device sector size, - then returned a buffer of that ROUNDED length, - which the caller compared via SequenceEqual against the original `entry` byte[] (length 1024). When SectorSize was 512 (the old constant probe), 1024 rounded to 1024 and the lengths happened to match. With the new max(logical, physical) probe returning 4096 on 4Kn drives (Windows/Ubuntu CI runners), 1024 rounds to 4096, the returned buffer is 4096 bytes long, and SequenceEqual fails on length mismatch (regardless of content). Pre-existing latent bug — the rounding to sector size is correct for the IO submit, but the caller should only see the bytes it asked for. Fix: return a buffer of the caller-requested logical `size`, not the sector-rounded `numBytesToRead`. Verification: 64/64 Tsavorite.test.hlog NativeDeviceTest + IDevice + NativeStorageDevice tests pass on Linux (where SectorSize is 4096 on the CI runner's 4Kn drive). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite benchmarks: refresh stale completion-threads help text The --device-completion-threads (KV.benchmark) and --completion-threads (Device.benchmark) help text said "all drainers share the same kernel io_context / io_uring" and "values > 1 are rarely useful past 1 today". Both claims are stale since the sharded-rings work (8cbca9d4d) and the per-thread ring affinity + 4-default-rings change (298bfd180): - Each drainer is bound 1:1 to its own kernel io_context (libaio) or io_uring ring (uring). - Submitters distribute across rings via per-thread affinity. - For io_uring, throughput scales with completion-threads up to available submitter concurrency (measured: ct=1 ~340K → ct=4 ~735K on Dell P5600 NVMe at the device-benchmark level). - For libaio extra drainers still rarely help past 1 (kernel per-context mutex is efficient). - Note added that uring uses min 4 rings even at ct=1 with the single drainer covering all rings via the legacy QueueRun scanner. Help-text-only change. No code behavior change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tsavorite Native: fix KV.benchmark deadlock on multi-segment disk-spill reads Root cause: cross-segment read rejection + engine retry loop ============================================================== The AllocatorBase.GetAndPopulateReadBuffer sector-aligned read window can extend past the page-end boundary when reading a record near the tail of a page. When the device's segment size is a multiple of the page size (e.g. 4MB pages, 1GB segments — the Garnet default), an over-extended read at the last page of a segment also crosses the device's segment boundary. NativeStorageDevice's underlying FileSystemSegmentedFile rejects cross-segment reads with Status::IOError; the engine's AsyncGetFromDiskCallback interprets a 0-byte read as a short read and retries the same address — forever. Worker thread spins at 99% CPU, disk activity drops to zero, benchmark deadlocks. Reproduced reliably on KV.benchmark: --device native --device-io-backend libaio \ --log-memory 16m --page-size 4m --segment-size 1g \ -n 10000000 (1.28 GB dataset → crosses 1GB segment boundary) Smaller datasets (1M = 128MB, fits in 1 segment) work; larger ones hang. RandomAccess device works on all dataset sizes because its managed segmented-file wrapper doesn't reject cross-segment reads. Diagnostic captured the exact symptom: a read at sourceAddress 0x3FFFF600 (1,073,739,776 — 2,560 bytes before the 1GB segment boundary) with readLength 4608 (sector-aligned record window) extends to 0x40000C00 — 2,560 bytes into segment 1. Native rejects with Status::IOError, callback fires with numBytes=0, engine retries. Fix: clamp the aligned read length so it never crosses page-end. ============================================================ Added in AllocatorBase.GetAndPopulateReadBuffer: var pageEndInFile = (ulong)(AlignedPageSizeBytes * (GetPage(fromLogicalAddress) + 1)); if (alignedFileOffset + alignedReadLength > pageEndInFile) alignedReadLength = (uint)(pageEndInFile - alignedFileOffset); Records never span page boundaries (HandlePageOverflow guarantees), so the actual record is fully readable within the clamped window — available_bytes reflects what we actually got from disk, and the engine continues normally. pageEnd is sector-aligned (PageSizeBits >= sector size), so the clamped length stays sector-aligned. Also reverted the uring "min 4 rings even at ct=1" experiment ============================================================= The earlier "default 4 rings for uring regardless of ct" change was fundamentally broken: with per-thread submit affinity (pick_ring's thread_local index), submitters bound to rings 1-3 never get their completions drained because the single drainer blocks on ring 0 with a 1-second QueueRun timeout and only briefly polls the other rings between wake-ups. The result is ~50x throughput degradation on workloads where submitters land on rings != 0 (KV.benchmark load phase dropped from 2.5M ops/sec to 54K ops/sec at t=1). Reverted to the simple rule: rings == numCompletionThreads. For uring perf scaling, users set numCompletionThreads >= expected submitter concurrency; each ring is then continuously drained by its dedicated drainer thread. Defense-in-depth hardening ========================== - NativeStorageDevice._callback now catches ALL exceptions from the user callback (was: try/finally but exception propagated). A managed exception escaping back into native code across the C ABI boundary silently terminates the drainer thread; the next submitter then spins forever in device.Throttle(). Now the exception is logged and swallowed so the drainer survives. - NativeStorageDevice.CompletionWorker has the same try/catch around the whole drain loop as defense-in-depth against unrelated managed exceptions (P/Invoke marshalling, IntPtr.Zero races with Dispose, etc.). - file_linux.cc QueueFile::ScheduleOperation (libaio) and UringFile::ScheduleOperation (uring) now retry submit-side EAGAIN indefinitely with bounded backoff (64 sched_yields, then 1ms nanosleeps) instead of returning Status::IOError after 8 yields. Surfacing transient EAGAIN as a permanent error creates the same retry-loop pathology as the cross-segment-read bug above. EAGAIN is the kernel saying "ring is full, try later"; it's not a real error and must not be exposed to the engine. Verification ============ KV.benchmark, 100M keys × 100B, 16MB log (mostly disk-spill), 1 completion thread, 100% reads: libaio: t=1 135K ops/sec, t=4 400K, t=8 444K, t=16 445K, t=32 404K uring: t=1 124K ops/sec, t=4 244K, t=8 265K, t=16 278K, t=32 272K Both backends stable across the full thread × dataset sweep (previously native+libaio hung on any 10M+ dataset; native+uring hung on every config). 64/64 Tsavorite.test.hlog IDevice + NativeStorageDevice tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> | 3 个月前 | |
Clear the Vector Set index pointer on the diskless replication stream-in path (#2043) * Clear the Vector Set index pointer on the diskless replication stream-in path A Vector Set index record persists the primary's native DiskANN handle (IndexPtr) inside its value. Diskless sync streams log records verbatim into the replica's memory, so the replica receives the primary's pointer. The only hook that zeroes that field is GarnetRecordTriggers.OnDiskRead, which by definition never fires for records streamed straight into memory. Since VectorManager.NeedsRecreate is exactly indexPtr == 0, a foreign pointer is indistinguishable from a healthy local one: the lazy Service.RecreateIndex rebuild is skipped and the raw value reaches the P/Invoke unvalidated, faulting the replica with SIGSEGV inside NativeDiskANNMethods.card. NetworkClusterSync now clears the pointer as records are deserialized, so the replica rebuilds its own index on first touch. Also rebuilds the context reservation after a diskless full sync. Such a sync SETs the streamed index and ContextMetadata records straight into the store, bypassing the RMW path that maintains the in-memory context-reservation bitmap. Unlike startup recovery, nothing rebuilt it, so after promotion a fresh Vector Set could be handed a context a streamed set already owns, corrupting both. Streamed records are fed through SanitizeAndTrackIngestedRecordIfApplicable during CLUSTER SYNC, and ReconcileRecoveredState rebuilds the reservation once the stream completes. That call passes requireNoReservedContexts on the diskless path: the rebuild replaces contextMetadatas wholesale, so a context surviving the preceding flush would be dropped without its index cleaned up. Adds Garnet.test.cluster.replication.vectorsets, registered in Garnet.slnx and in the CI and nightly matrices. The harness asserts index ownership and element-level equality, since matching cardinality alone is too weak: an aliased index reports the source's count perfectly well. Covers diskless full sync, async replay, multi-database sync, exception-injected aborted syncs, and FreshVectorSetDoesNotReuseStreamedContextAfterDisklessFullSync for the context-reservation regression. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3b280844-5962-4e83-9a36-c9f9b64639b0 * Document the default-namespace encoding in SanitizeAndTrackIngestedRecordIfApplicable A record with no namespace reports a single 0 byte rather than an empty span, so the ContextMetadata check must be a positive test for MetadataNamespace. Testing for a non-empty namespace instead routes index records away from ClearIndexPointer, which reintroduces the foreign IndexPtr fault this change fixes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3b280844-5962-4e83-9a36-c9f9b64639b0 * Say 'reserved contexts' to match requireNoReservedContexts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3b280844-5962-4e83-9a36-c9f9b64639b0 * Guard the namespace check with HasNamespace ISourceLogRecord extends IKey, so HasNamespace/NamespaceBytes are available on the generic constraint. This restores the shape used by the recovery path. Note that testing the namespace span for emptiness is not equivalent: a record without a namespace reports a single 0 byte, which IKey documents as reserved. Treating that as 'namespaced' routes index records away from ClearIndexPointer and faults the replica. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3b280844-5962-4e83-9a36-c9f9b64639b0 * Note why recovered context metadata is saved off Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3b280844-5962-4e83-9a36-c9f9b64639b0 * Give the cluster test replication helpers more specific names Also null-guard the VectorManager reconcile on the diskless full sync path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5883e70d-95c1-49ec-981d-ff98afd48a5d * Derive Vector Set test seeds instead of hand-picking date constants PopulateVectorSet now derives its seed from the test name, key, and call ordinal via FNV-1a, so each vector set still gets distinct data that reproduces across runs without callers maintaining a numbering scheme. String.GetHashCode is randomized per process and cannot be used here. Also folds VectorSetsStayPartitionedAcrossDisklessFullSync into VectorSetReadableOnReplicaAfterDisklessFullSync, which already covers the same sync path and now carries the multi-set cardinality isolation checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5883e70d-95c1-49ec-981d-ff98afd48a5d * Drop the seed from the Vector Set replication test harness Embeddings are read back from both nodes and compared at runtime rather than regenerated, so the assertions never depend on the vector data being reproducible. Successive Random instances still yield distinct data per vector set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5883e70d-95c1-49ec-981d-ff98afd48a5d * Drop the migration test from the diskless sync fixture Migration transfers Vector Sets through its own path: the destination reserves contexts via CLUSTER RESERVE VECTOR_SET_CONTEXTS and the index record is remapped with SetContextForMigration before it is sent. SanitizeAndTrackIngestedRecordIfApplicable is only reached from CLUSTER SYNC stream-in and recovery, so no record in that test crossed the code this fixture covers. Vector Set migration is already covered by ClusterVectorSetTests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5883e70d-95c1-49ec-981d-ff98afd48a5d * Add a scoped exception injection helper and use it in the diskless sync tests The injection flags are global statics, so a test that fails to disable one leaks it into everything that runs after it. ExceptionInjectionHelper.Enabled returns a struct that disables on dispose, replacing the hand written try/finally pairs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5883e70d-95c1-49ec-981d-ff98afd48a5d * Drop the redundant WaitUntilServes waits from the diskless sync tests AssertFullyReplicated already waits for the replica AOF offset to catch up, which the replica only publishes after ReconcileRecoveredState completes, so the extra key-visibility poll added nothing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5883e70d-95c1-49ec-981d-ff98afd48a5d * Correct a comment: contexts are reserved in the VectorManager, not an allocator Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5883e70d-95c1-49ec-981d-ff98afd48a5d * Rename ExceptionInjectionHelper.Enabled to EnabledScope Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5883e70d-95c1-49ec-981d-ff98afd48a5d * Trim the EnabledScope doc comment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5883e70d-95c1-49ec-981d-ff98afd48a5d * Guard the flush-time dirty context reset against uninitialized databases FLUSHALL walks every active database, and a database that was only ever created by SELECT has a VectorManager whose Initialize has not run, so dirtyContextMetadatas is still null and clearing it killed the session. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5883e70d-95c1-49ec-981d-ff98afd48a5d --------- Co-authored-by: tiagonapoli <tiagonapoli@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3b280844-5962-4e83-9a36-c9f9b64639b0 Copilot-Session: 5883e70d-95c1-49ec-981d-ff98afd48a5d | 27 天前 | |
Initial commit | 2 年前 | |
Initial commit | 2 年前 | |
JSON Module rewrite with custom JSONPath implementation (#974) * Initial commit of Json Module rewrite * Changed Test to use JsonNode directly * Implementation of JSON.GET and SET commands * Moved other moduels from Json BDN * Added more benchmark * Moved json modulke to root location * Updated Licenses * Added documentation * Code cleanup and code comments * Fixed code format * Fixed build issue * Added more test cases * Added more test cases * Fix formating * Add expected values for ACL and AOF operations in benchmark config * Fixed test case failure * Review comment fixes * Fxied format issue * Fixed review comments * Fixed review comments and memory changes --------- Co-authored-by: Tal Zaccai <talzacc@microsoft.com> Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> | 1 年前 | |
Update docs, move purgebp to a debug subcommand (#1984) * Update documentation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 513c86bf-84d2-4521-88c6-c963e28036b0 * Move PURGEBP to a DEBUG subcommand PURGEBP is a debug/admin utility, so move it from a top-level RESP command to `DEBUG PURGEBP <manager-type>`, alongside FORCEGC/FLUSHANDEVICT. The read-only top-level RespCommand.PURGEBP is removed; its frozen v3 AOF slot maps to NONE (never persisted). Metadata, ACL test, and docs updated accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 513c86bf-84d2-4521-88c6-c963e28036b0 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Ted Hart <15467143+TedHartMS@users.noreply.github.com> Copilot-Session: 513c86bf-84d2-4521-88c6-c963e28036b0 | 1 个月前 | |
Initial commit | 2 年前 | |
Fix link in SUPPORT.md and adding vscode and rider files to gitignore file (#48) Co-authored-by: darrenge <darrenge@microsoft.com> | 2 年前 | |
Bump version to 2.1.5 (#2084) Co-authored-by: Tiago Martins Napoli <tiagonapoli@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> | 12 天前 | |
Update docker-compose to use latest official images (#183) * Update docker-compose.yml | 2 年前 | |
Add es-metadata.yml (#1641) | 5 个月前 | |
Build(deps): Bump dotnet-sdk in the dotnet-deps group (#2004) Bumps the dotnet-deps group with 1 update: [dotnet-sdk](https://github.com/dotnet/sdk). Updates `dotnet-sdk` from 10.0.301 to 10.0.302 - [Release notes](https://github.com/dotnet/sdk/releases) - [Commits](https://github.com/dotnet/sdk/compare/v10.0.301...v10.0.302) --- updated-dependencies: - dependency-name: dotnet-sdk dependency-version: 10.0.302 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dotnet-deps ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> | 1 个月前 |
以下内容由 AI 翻译，如有问题请 点此提交 issue 反馈
Garnet
Garnet 是一款远程缓存存储，具有多项独特优势：
- Garnet 采用广受欢迎的 RESP 有线协议作为起点，这使得 Garnet 能够直接使用当今大多数编程语言中现成的 Redis 客户端，例如 C# 中的 StackExchange.Redis。
- 相比同类开源缓存存储，Garnet 在处理大量客户端连接和小批量数据时，展现出更卓越的吞吐量和可扩展性，能为大型应用和服务节省成本。
- 在启用了加速网络的通用云（Azure）虚拟机上，Garnet 可实现极低的客户端延迟（99.9% 分位通常低于 300 微秒），这对实际业务场景至关重要。
- Garnet 基于最新的 .NET 技术构建，具备跨平台、可扩展和现代化的特性。其设计旨在便于开发和迭代，同时不影响常见场景下的性能。我们借助 .NET 丰富的库生态系统来拓展 API 广度，并为优化预留了开放空间。通过对 .NET 的精心运用，Garnet 在 Linux 和 Windows 平台上均实现了业界领先的性能。
本代码库包含构建和运行 Garnet 的代码。如需更多信息和文档，请访问我们的网站：https://microsoft.github.io/garnet。
正在寻找完全托管的服务？ Azure Cosmos DB Garnet Cache 提供 Garnet 作为完全托管、企业级的缓存解决方案，内置高可用性、性能保障和零基础设施管理。
新特性一览 🎉
- 🚀 向量集合（预览版） — 基于 DiskANN 算法和 Garnet 的 Tsavorite 存储引擎，提供近似最近邻搜索功能。在我们的 初步结果 中，Garnet 在每秒查询数（QPS）、p99 延迟和召回率方面均处于领先地位。
- 🔍 范围索引（预览版） — 借助 Bf-Tree 技术，为 Garnet 键提供二级范围索引和等值索引。
- 📄 Garnet 论文将在 VLDB 2026 会议上发表！ B. Chandramouli, V. Zois, T. Hart, T. Zaccai, L. M. Maas, Y. Rajasekaran, D. Gehring. Garnet: A Next-Generation Cache-Store for Accelerating Applications and Services. PVLDB 2026. [PDF]
功能概述
Garnet 实现了广泛的 API，包括原始字符串操作（如 gets、sets 和键过期）、分析型操作（如 HyperLogLog 和 Bitmap）以及对象型操作（如有序集合和列表）。它能够处理多键事务，支持客户端 RESP 事务以及我们自研的 C# 服务器端存储过程和模块。用户可以在 C# 环境中，以便捷且安全的方式，为原始字符串和自定义对象类型定义自定义操作，从而降低了开发自定义扩展的门槛。Garnet 同时支持 Lua 脚本。
Garnet 采用快速且可插拔的网络层，为未来扩展（如利用内核绕过栈）奠定了基础。它使用 .NET 强大的 SslStream 库支持安全的传输层安全（TLS）通信，并提供基本的访问控制功能。Garnet 的存储层名为 Tsavorite，专为高性能而构建，具备强大的数据库特性，例如线程可扩展性、分层存储支持（内存、SSD 和云存储）、快速无阻塞检查点、恢复机制、用于持久性的操作日志、多键事务支持以及更优的内存管理与复用。最后，Garnet 支持集群模式运行，具备分片、复制和动态键迁移功能。
性能预览
我们在官方网站上展示了一些关键结果，将 Garnet 与主流开源缓存存储进行了对比。
设计亮点
Garnet 的设计对整个缓存存储栈进行了重新思考——从网络上接收数据包，到解析和处理数据库操作，再到执行存储交互。我们的设计建立在多年先前研究的基础之上。以下是 Garnet 的整体架构。
Garnet 的网络层基于共享内存设计，TLS 处理和存储交互在网络 IO 完成线程上执行，避免了常见情况下的线程切换开销。这种方法利用 CPU 缓存一致性将数据传输到处理逻辑，而非传统的基于 shuffle 的网络设计，后者需要将数据移动到服务器上的相应分片。
Garnet 的存储设计包含两个 Tsavorite 键值存储，它们的命运由一个统一的操作日志绑定。第一个存储称为“主存储”，针对原始字符串操作进行了优化，并对内存进行精心管理以避免垃圾回收。第二个（可选的）“对象存储”针对复杂对象和自定义数据类型进行了优化，包括 Sorted Set、Set、Hash、List 和 Geo 等常用类型。对象存储中的数据类型在当前实现中利用了 .NET 库生态系统。它们在内存中以堆的形式存储（这使得更新非常高效），并在磁盘上以序列化形式存储。未来，我们计划研究使用统一的索引和日志以简化维护。
Garnet 设计的一个显著特点是其窄腰型 Tsavorite 存储 API，该 API 用于在其之上实现庞大、丰富且可扩展的 RESP API 表面。此 API 包括读取、更新插入、删除和原子性读写操作，并通过异步回调实现，使 Garnet 能够在每个操作的各个阶段插入逻辑。我们的存储 API 模型使我们能够将 Garnet 的解析和查询处理关注点与并发、存储分层和 checkpointing 等存储细节清晰地分离。Garnet 对多键事务使用两阶段锁定。
集群模式
除了单节点运行外，Garnet 还具备功能完备的集群模式，支持用户创建和管理分片与复制部署。Garnet 同样支持高效且动态的键迁移方案，用于重新平衡分片。用户可使用标准的 Redis 集群命令来创建和管理 Garnet 集群，节点通过 gossip 协议共享和更新集群状态。Garnet 的集群模式设计目前为被动式：这意味着它不实现领导者选举，仅响应用户提供的控制平面所发出的集群命令；详情请参见此链接。
后续步骤
许可证
隐私
隐私信息请参见 https://privacy.microsoft.com/en-us/。
贡献
本项目欢迎各类贡献和建议。大多数贡献要求您同意《贡献者许可协议》(CLA)，以声明您有权并实际授予我们使用您贡献的权利。详情请访问 https://cla.opensource.microsoft.com。
当您提交拉取请求时，CLA 机器人将自动判断您是否需要提供 CLA，并对 PR 进行相应标记（例如状态检查、评论）。您只需按照机器人提供的指示操作即可。在所有使用我们 CLA 的仓库中，您只需完成一次此操作。
本项目已采用 Microsoft 开源行为准则。如需了解更多信息，请参见行为准则常见问题，或通过 opencode@microsoft.com 联系我们以获取其他问题或意见的解答。
商标
本项目可能包含相关项目、产品或服务的商标或徽标。Microsoft 商标或徽标的授权使用受 Microsoft 商标与品牌指南 约束，且必须遵循该指南。
在本项目的修改版本中使用 Microsoft 商标或徽标时，不得造成混淆，也不得暗示 Microsoft 的赞助。
任何第三方商标或徽标的使用均受该第三方政策的约束。
Redis 是 Redis Ltd. 的注册商标。其所有权利归 Redis Ltd. 所有。Microsoft 对 Redis 的任何使用仅为参考目的，并不表示 Redis 与 Microsoft 之间存在任何赞助、背书或关联关系。
项目介绍
Garnet is a remote cache-store from Microsoft Research that offers strong performance (throughput and latency), scalability, storage, recovery, cluster sharding, key migration, and replication features. Garnet can work with existing Redis clients.
MIT C#1.35 K提交数cachecache-storageclusterconcurrenthash-tablekey-valuelarger-than-memorylow-latencypersistentremotescalablestore
定制我的领域