| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Allocate test ports dynamically from a non-ephemeral range (#2193) * Refuse a second bind on an occupied port on Windows, and add an avalanched hash On Windows GarnetServerTcp set SO_REUSEADDR unconditionally, and cleared SO_REUSEPORT only on Linux, macOS, and FreeBSD. Windows SO_REUSEADDR is permissive: it lets a socket bind an address and port another socket already holds. Two live servers therefore both bound the same port and the OS split incoming connections between them, so a port conflict surfaced as plausible but wrong results rather than as a bind failure. Windows now sets SO_EXCLUSIVEADDRUSE, which refuses the second bind; the Unix paths are unchanged. GarnetServerTcpTests covered this but was skipped on Windows because the behavior it asserts did not hold there. It now runs on every platform, and its TIME_WAIT rebind case confirms exclusive use does not block restarts. Utility.GetHashCodeWithMix applies a MurmurHash3 fmix64 finalizer to HashBytes. HashBytes is a multiply-accumulate, and multiplication propagates information only toward higher bits, so its low bits carry almost no entropy: flipping one input bit changes some output bits with probability 1.000 rather than 0.5. The finalizer's right shifts move entropy back down, which makes it safe to reduce the result modulo a power of two. UtilityTests asserts the avalanche property directly, since that, rather than the hash value, is why the method exists. * Allocate test ports dynamically from a non-ephemeral range Test ports were fixed per sub-project from a base of 33278, shifted by an optional GARNET_TEST_PORT_SLOT offset. That base predates the slot mechanism and sits inside Linux's default outbound-connection range (32768-60999, net.ipv4.ip_local_port_range), so on Linux the kernel could assign a test's port to a connect() between the moment the port was observed free and the moment the server bound it. Probing cannot close that window; only staying out of the range can. The slot mechanism encoded 49152 as the ephemeral floor, which is the Windows and macOS value, so slot 7 looked like the boundary case while slot 0 was already unsafe on Linux. Windows and macOS were never affected, which is why the flakes were Linux-only. Ports now come from [16384, 32768), below every supported platform's ephemeral range and above the crowded region that holds both ports a developer is likely to be running (6379, 5432, 11211, 8080) and ports this suite itself uses (10000-10002 Azurite). Each test host leases one aligned 32-port block, held by an exclusively locked file for the life of the process, which is what keeps concurrent hosts apart; the OS releases the lock on exit, crash, or kill, so a block frees itself. The lease is taken first and the ports probed second, because software outside the suite, or a server stranded by a killed run, holds ports while holding no lease. Blocks are searched from a start derived from the checkout and the project name, so a project returns to the same ports run after run and netstat output stays interpretable. On collision the search advances by a stride derived from the process, so two hosts that start at the same block diverge rather than walking in lockstep. The stride is forced odd and the block count is a power of two, which together make the walk visit every block exactly once: completing it proves no block is free rather than exhausting a retry budget. The hash is Utility.GetHashCodeWithMix, because the index is cut from the low bits that HashBytes alone leaves unmixed. This replaces slots entirely. Concurrency is no longer capped at seven checkouts, GARNET_TEST_PORT_SLOT is gone and nothing needs setting, and TestPortAssignment, ClusterPortAssignment, and the band-width invariants they required are deleted - projects are identified by assembly name, so adding one needs no central registration. Special cases, each commented where it lives: - Ports 30000-30999 are carved out for test/docker-tests/validate_docker_images.py, which allocates from its own fixed base and can run alongside this suite. TestPortAllocatorTests reads that base out of the Python file so moving either side without the other fails as a test. - The probe binds the IPv4 and IPv6 wildcards separately and exclusively. One dual-mode socket handles IPv4-mapped addresses differently per platform, and a shared bind succeeds alongside a listener on a specific address under that wildcard. - A failed probe deliberately ignores the socket error code: Windows reports AccessDenied for ports excluded by Hyper-V, WSL, or Docker, not AddressAlreadyInUse. - Cluster nodes keep contiguous ports because node n binds base + n; block alignment keeps that run from straddling a boundary. - GARNET_TEST_PORT_BASE pins a base port for reproduction or a fixed firewall opening. Pinned ports are still leased and probed. GarnetServerTcp.Start now reports the endpoint and the operating system's reason when a bind fails, which the raw SocketException names neither of. TestUtils.OnTearDown adds the allocation context, so a conflict reads as software outside the suite rather than as a defect in Garnet or in the test. * Drop stale port-slot instructions from the add-garnet-command skill The skill still told readers to set GARNET_TEST_PORT_SLOT, which no longer exists; test hosts now claim their ports without configuration. * Cover the port allocator's failure paths Reserve's error paths decide whether a run stops clearly or proceeds on ports it does not own, and none of them ran in an ordinary suite: a block another host holds, a pinned base port, and every block exhausted. A regression in any of them would surface as the silent cross-checkout corruption the allocator exists to prevent. Leases in these tests are redirected to a private directory. Holding blocks in the shared one would stop concurrent runs in other checkouts from starting, which is the very failure under test. Writing them found a real defect in the first draft: two tests took the same port from a deterministic helper, and because a reservation holds its lease for the life of the process while binding nothing, the second saw ports that read as free inside a block that was already leased. The helper now checks both, which is the same distinction the allocator draws. BuildPortConflictReport is split out of ReportPortConflictIfBindFailed so the decision and the wording can be asserted without making a real test fail. * Name the alternate port explicitly in ConnectionLimitAcrossListenersTests The fixture built its second listener from TestUtils.TestPort + 1. That is the port TestUtils.AlternateTestPort returns, and it is inside the block this test host leases, so the behavior is unchanged - but the arithmetic only works because the reservation covers two ports, which the expression does not say. Naming the property states the dependency the fixture actually has. * Fix overlapping pinned reservations and misleading bind diagnostics Review of the allocator found four defects, all in paths that only run when something has already gone wrong. ReservePinned leased only the block containing its first port. A pinned base is deliberately not block-aligned, so a run can cross into the next block: base 16415 with two ports leases block 0 while port 16416 sits unleased in block 1. Another host could lease that block and be handed an overlapping run, both probes passing because neither side had bound yet. Every intersected block is now leased. IntersectedBlocks also skips ports outside the allocatable range, where PortBlock is not meaningful - it returned -480 for port 1024 and 738 for port 40000, neither of which names a block. The teardown report fired for every bind failure but always diagnosed a port conflict. GarnetServerTcp also emits "Could not bind" for AccessDenied, which is a platform-reserved range, and AddressNotAvailable, which is an address absent from the machine; neither is another process taking the port. The report now parses the socket error and stays silent unless it is AddressAlreadyInUse. The report read the standalone port field, which ClusterTestContext.ReservePorts never sets, so every cluster bind failure claimed the host had reserved no ports while it held an eight-port run. The allocator now records each reservation and the report looks up the run covering the failing port, which also lets it name the project that holds it. The report asserted that a reserved port cannot have been taken by the OS. That is true only inside the allocatable range. GARNET_TEST_PORT_BASE can pin a run above it, where the kernel really can assign the port to an outbound connection between the probe and the bind - the one collision pinning opts into. Pinned runs outside the range are now described as exactly that, and an endpoint the test built itself is reported as carrying no allocator guarantee at all. Each fix is covered by a test verified by reverting the fix: restoring the single-block lease reports the tail of a straddling run as still available, and removing the error-code gate reports an AccessDenied failure as software outside the suite. | 27 天前 | |
Allocate test ports dynamically from a non-ephemeral range (#2193) * Refuse a second bind on an occupied port on Windows, and add an avalanched hash On Windows GarnetServerTcp set SO_REUSEADDR unconditionally, and cleared SO_REUSEPORT only on Linux, macOS, and FreeBSD. Windows SO_REUSEADDR is permissive: it lets a socket bind an address and port another socket already holds. Two live servers therefore both bound the same port and the OS split incoming connections between them, so a port conflict surfaced as plausible but wrong results rather than as a bind failure. Windows now sets SO_EXCLUSIVEADDRUSE, which refuses the second bind; the Unix paths are unchanged. GarnetServerTcpTests covered this but was skipped on Windows because the behavior it asserts did not hold there. It now runs on every platform, and its TIME_WAIT rebind case confirms exclusive use does not block restarts. Utility.GetHashCodeWithMix applies a MurmurHash3 fmix64 finalizer to HashBytes. HashBytes is a multiply-accumulate, and multiplication propagates information only toward higher bits, so its low bits carry almost no entropy: flipping one input bit changes some output bits with probability 1.000 rather than 0.5. The finalizer's right shifts move entropy back down, which makes it safe to reduce the result modulo a power of two. UtilityTests asserts the avalanche property directly, since that, rather than the hash value, is why the method exists. * Allocate test ports dynamically from a non-ephemeral range Test ports were fixed per sub-project from a base of 33278, shifted by an optional GARNET_TEST_PORT_SLOT offset. That base predates the slot mechanism and sits inside Linux's default outbound-connection range (32768-60999, net.ipv4.ip_local_port_range), so on Linux the kernel could assign a test's port to a connect() between the moment the port was observed free and the moment the server bound it. Probing cannot close that window; only staying out of the range can. The slot mechanism encoded 49152 as the ephemeral floor, which is the Windows and macOS value, so slot 7 looked like the boundary case while slot 0 was already unsafe on Linux. Windows and macOS were never affected, which is why the flakes were Linux-only. Ports now come from [16384, 32768), below every supported platform's ephemeral range and above the crowded region that holds both ports a developer is likely to be running (6379, 5432, 11211, 8080) and ports this suite itself uses (10000-10002 Azurite). Each test host leases one aligned 32-port block, held by an exclusively locked file for the life of the process, which is what keeps concurrent hosts apart; the OS releases the lock on exit, crash, or kill, so a block frees itself. The lease is taken first and the ports probed second, because software outside the suite, or a server stranded by a killed run, holds ports while holding no lease. Blocks are searched from a start derived from the checkout and the project name, so a project returns to the same ports run after run and netstat output stays interpretable. On collision the search advances by a stride derived from the process, so two hosts that start at the same block diverge rather than walking in lockstep. The stride is forced odd and the block count is a power of two, which together make the walk visit every block exactly once: completing it proves no block is free rather than exhausting a retry budget. The hash is Utility.GetHashCodeWithMix, because the index is cut from the low bits that HashBytes alone leaves unmixed. This replaces slots entirely. Concurrency is no longer capped at seven checkouts, GARNET_TEST_PORT_SLOT is gone and nothing needs setting, and TestPortAssignment, ClusterPortAssignment, and the band-width invariants they required are deleted - projects are identified by assembly name, so adding one needs no central registration. Special cases, each commented where it lives: - Ports 30000-30999 are carved out for test/docker-tests/validate_docker_images.py, which allocates from its own fixed base and can run alongside this suite. TestPortAllocatorTests reads that base out of the Python file so moving either side without the other fails as a test. - The probe binds the IPv4 and IPv6 wildcards separately and exclusively. One dual-mode socket handles IPv4-mapped addresses differently per platform, and a shared bind succeeds alongside a listener on a specific address under that wildcard. - A failed probe deliberately ignores the socket error code: Windows reports AccessDenied for ports excluded by Hyper-V, WSL, or Docker, not AddressAlreadyInUse. - Cluster nodes keep contiguous ports because node n binds base + n; block alignment keeps that run from straddling a boundary. - GARNET_TEST_PORT_BASE pins a base port for reproduction or a fixed firewall opening. Pinned ports are still leased and probed. GarnetServerTcp.Start now reports the endpoint and the operating system's reason when a bind fails, which the raw SocketException names neither of. TestUtils.OnTearDown adds the allocation context, so a conflict reads as software outside the suite rather than as a defect in Garnet or in the test. * Drop stale port-slot instructions from the add-garnet-command skill The skill still told readers to set GARNET_TEST_PORT_SLOT, which no longer exists; test hosts now claim their ports without configuration. * Cover the port allocator's failure paths Reserve's error paths decide whether a run stops clearly or proceeds on ports it does not own, and none of them ran in an ordinary suite: a block another host holds, a pinned base port, and every block exhausted. A regression in any of them would surface as the silent cross-checkout corruption the allocator exists to prevent. Leases in these tests are redirected to a private directory. Holding blocks in the shared one would stop concurrent runs in other checkouts from starting, which is the very failure under test. Writing them found a real defect in the first draft: two tests took the same port from a deterministic helper, and because a reservation holds its lease for the life of the process while binding nothing, the second saw ports that read as free inside a block that was already leased. The helper now checks both, which is the same distinction the allocator draws. BuildPortConflictReport is split out of ReportPortConflictIfBindFailed so the decision and the wording can be asserted without making a real test fail. * Name the alternate port explicitly in ConnectionLimitAcrossListenersTests The fixture built its second listener from TestUtils.TestPort + 1. That is the port TestUtils.AlternateTestPort returns, and it is inside the block this test host leases, so the behavior is unchanged - but the arithmetic only works because the reservation covers two ports, which the expression does not say. Naming the property states the dependency the fixture actually has. * Fix overlapping pinned reservations and misleading bind diagnostics Review of the allocator found four defects, all in paths that only run when something has already gone wrong. ReservePinned leased only the block containing its first port. A pinned base is deliberately not block-aligned, so a run can cross into the next block: base 16415 with two ports leases block 0 while port 16416 sits unleased in block 1. Another host could lease that block and be handed an overlapping run, both probes passing because neither side had bound yet. Every intersected block is now leased. IntersectedBlocks also skips ports outside the allocatable range, where PortBlock is not meaningful - it returned -480 for port 1024 and 738 for port 40000, neither of which names a block. The teardown report fired for every bind failure but always diagnosed a port conflict. GarnetServerTcp also emits "Could not bind" for AccessDenied, which is a platform-reserved range, and AddressNotAvailable, which is an address absent from the machine; neither is another process taking the port. The report now parses the socket error and stays silent unless it is AddressAlreadyInUse. The report read the standalone port field, which ClusterTestContext.ReservePorts never sets, so every cluster bind failure claimed the host had reserved no ports while it held an eight-port run. The allocator now records each reservation and the report looks up the run covering the failing port, which also lets it name the project that holds it. The report asserted that a reserved port cannot have been taken by the OS. That is true only inside the allocatable range. GARNET_TEST_PORT_BASE can pin a run above it, where the kernel really can assign the port to an outbound connection between the probe and the bind - the one collision pinning opts into. Pinned runs outside the range are now described as exactly that, and an endpoint the test built itself is reported as carrying no allocator guarantee at all. Each fix is covered by a test verified by reverting the fix: restoring the single-block lease reports the tail of a straddling run as still available, and removing the error-code gate reports an AccessDenied failure as software outside the suite. | 27 天前 | |
Allocate test ports dynamically from a non-ephemeral range (#2193) * Refuse a second bind on an occupied port on Windows, and add an avalanched hash On Windows GarnetServerTcp set SO_REUSEADDR unconditionally, and cleared SO_REUSEPORT only on Linux, macOS, and FreeBSD. Windows SO_REUSEADDR is permissive: it lets a socket bind an address and port another socket already holds. Two live servers therefore both bound the same port and the OS split incoming connections between them, so a port conflict surfaced as plausible but wrong results rather than as a bind failure. Windows now sets SO_EXCLUSIVEADDRUSE, which refuses the second bind; the Unix paths are unchanged. GarnetServerTcpTests covered this but was skipped on Windows because the behavior it asserts did not hold there. It now runs on every platform, and its TIME_WAIT rebind case confirms exclusive use does not block restarts. Utility.GetHashCodeWithMix applies a MurmurHash3 fmix64 finalizer to HashBytes. HashBytes is a multiply-accumulate, and multiplication propagates information only toward higher bits, so its low bits carry almost no entropy: flipping one input bit changes some output bits with probability 1.000 rather than 0.5. The finalizer's right shifts move entropy back down, which makes it safe to reduce the result modulo a power of two. UtilityTests asserts the avalanche property directly, since that, rather than the hash value, is why the method exists. * Allocate test ports dynamically from a non-ephemeral range Test ports were fixed per sub-project from a base of 33278, shifted by an optional GARNET_TEST_PORT_SLOT offset. That base predates the slot mechanism and sits inside Linux's default outbound-connection range (32768-60999, net.ipv4.ip_local_port_range), so on Linux the kernel could assign a test's port to a connect() between the moment the port was observed free and the moment the server bound it. Probing cannot close that window; only staying out of the range can. The slot mechanism encoded 49152 as the ephemeral floor, which is the Windows and macOS value, so slot 7 looked like the boundary case while slot 0 was already unsafe on Linux. Windows and macOS were never affected, which is why the flakes were Linux-only. Ports now come from [16384, 32768), below every supported platform's ephemeral range and above the crowded region that holds both ports a developer is likely to be running (6379, 5432, 11211, 8080) and ports this suite itself uses (10000-10002 Azurite). Each test host leases one aligned 32-port block, held by an exclusively locked file for the life of the process, which is what keeps concurrent hosts apart; the OS releases the lock on exit, crash, or kill, so a block frees itself. The lease is taken first and the ports probed second, because software outside the suite, or a server stranded by a killed run, holds ports while holding no lease. Blocks are searched from a start derived from the checkout and the project name, so a project returns to the same ports run after run and netstat output stays interpretable. On collision the search advances by a stride derived from the process, so two hosts that start at the same block diverge rather than walking in lockstep. The stride is forced odd and the block count is a power of two, which together make the walk visit every block exactly once: completing it proves no block is free rather than exhausting a retry budget. The hash is Utility.GetHashCodeWithMix, because the index is cut from the low bits that HashBytes alone leaves unmixed. This replaces slots entirely. Concurrency is no longer capped at seven checkouts, GARNET_TEST_PORT_SLOT is gone and nothing needs setting, and TestPortAssignment, ClusterPortAssignment, and the band-width invariants they required are deleted - projects are identified by assembly name, so adding one needs no central registration. Special cases, each commented where it lives: - Ports 30000-30999 are carved out for test/docker-tests/validate_docker_images.py, which allocates from its own fixed base and can run alongside this suite. TestPortAllocatorTests reads that base out of the Python file so moving either side without the other fails as a test. - The probe binds the IPv4 and IPv6 wildcards separately and exclusively. One dual-mode socket handles IPv4-mapped addresses differently per platform, and a shared bind succeeds alongside a listener on a specific address under that wildcard. - A failed probe deliberately ignores the socket error code: Windows reports AccessDenied for ports excluded by Hyper-V, WSL, or Docker, not AddressAlreadyInUse. - Cluster nodes keep contiguous ports because node n binds base + n; block alignment keeps that run from straddling a boundary. - GARNET_TEST_PORT_BASE pins a base port for reproduction or a fixed firewall opening. Pinned ports are still leased and probed. GarnetServerTcp.Start now reports the endpoint and the operating system's reason when a bind fails, which the raw SocketException names neither of. TestUtils.OnTearDown adds the allocation context, so a conflict reads as software outside the suite rather than as a defect in Garnet or in the test. * Drop stale port-slot instructions from the add-garnet-command skill The skill still told readers to set GARNET_TEST_PORT_SLOT, which no longer exists; test hosts now claim their ports without configuration. * Cover the port allocator's failure paths Reserve's error paths decide whether a run stops clearly or proceeds on ports it does not own, and none of them ran in an ordinary suite: a block another host holds, a pinned base port, and every block exhausted. A regression in any of them would surface as the silent cross-checkout corruption the allocator exists to prevent. Leases in these tests are redirected to a private directory. Holding blocks in the shared one would stop concurrent runs in other checkouts from starting, which is the very failure under test. Writing them found a real defect in the first draft: two tests took the same port from a deterministic helper, and because a reservation holds its lease for the life of the process while binding nothing, the second saw ports that read as free inside a block that was already leased. The helper now checks both, which is the same distinction the allocator draws. BuildPortConflictReport is split out of ReportPortConflictIfBindFailed so the decision and the wording can be asserted without making a real test fail. * Name the alternate port explicitly in ConnectionLimitAcrossListenersTests The fixture built its second listener from TestUtils.TestPort + 1. That is the port TestUtils.AlternateTestPort returns, and it is inside the block this test host leases, so the behavior is unchanged - but the arithmetic only works because the reservation covers two ports, which the expression does not say. Naming the property states the dependency the fixture actually has. * Fix overlapping pinned reservations and misleading bind diagnostics Review of the allocator found four defects, all in paths that only run when something has already gone wrong. ReservePinned leased only the block containing its first port. A pinned base is deliberately not block-aligned, so a run can cross into the next block: base 16415 with two ports leases block 0 while port 16416 sits unleased in block 1. Another host could lease that block and be handed an overlapping run, both probes passing because neither side had bound yet. Every intersected block is now leased. IntersectedBlocks also skips ports outside the allocatable range, where PortBlock is not meaningful - it returned -480 for port 1024 and 738 for port 40000, neither of which names a block. The teardown report fired for every bind failure but always diagnosed a port conflict. GarnetServerTcp also emits "Could not bind" for AccessDenied, which is a platform-reserved range, and AddressNotAvailable, which is an address absent from the machine; neither is another process taking the port. The report now parses the socket error and stays silent unless it is AddressAlreadyInUse. The report read the standalone port field, which ClusterTestContext.ReservePorts never sets, so every cluster bind failure claimed the host had reserved no ports while it held an eight-port run. The allocator now records each reservation and the report looks up the run covering the failing port, which also lets it name the project that holds it. The report asserted that a reserved port cannot have been taken by the OS. That is true only inside the allocatable range. GARNET_TEST_PORT_BASE can pin a run above it, where the kernel really can assign the port to an outbound connection between the probe and the bind - the one collision pinning opts into. Pinned runs outside the range are now described as exactly that, and an endpoint the test built itself is reported as carrying no allocator guarantee at all. Each fix is covered by a test verified by reverting the fix: restoring the single-block lease reports the tail of a straddling run as still available, and removing the error-code gate reports an AccessDenied failure as software outside the suite. | 27 天前 | |
Allocate test ports dynamically from a non-ephemeral range (#2193) * Refuse a second bind on an occupied port on Windows, and add an avalanched hash On Windows GarnetServerTcp set SO_REUSEADDR unconditionally, and cleared SO_REUSEPORT only on Linux, macOS, and FreeBSD. Windows SO_REUSEADDR is permissive: it lets a socket bind an address and port another socket already holds. Two live servers therefore both bound the same port and the OS split incoming connections between them, so a port conflict surfaced as plausible but wrong results rather than as a bind failure. Windows now sets SO_EXCLUSIVEADDRUSE, which refuses the second bind; the Unix paths are unchanged. GarnetServerTcpTests covered this but was skipped on Windows because the behavior it asserts did not hold there. It now runs on every platform, and its TIME_WAIT rebind case confirms exclusive use does not block restarts. Utility.GetHashCodeWithMix applies a MurmurHash3 fmix64 finalizer to HashBytes. HashBytes is a multiply-accumulate, and multiplication propagates information only toward higher bits, so its low bits carry almost no entropy: flipping one input bit changes some output bits with probability 1.000 rather than 0.5. The finalizer's right shifts move entropy back down, which makes it safe to reduce the result modulo a power of two. UtilityTests asserts the avalanche property directly, since that, rather than the hash value, is why the method exists. * Allocate test ports dynamically from a non-ephemeral range Test ports were fixed per sub-project from a base of 33278, shifted by an optional GARNET_TEST_PORT_SLOT offset. That base predates the slot mechanism and sits inside Linux's default outbound-connection range (32768-60999, net.ipv4.ip_local_port_range), so on Linux the kernel could assign a test's port to a connect() between the moment the port was observed free and the moment the server bound it. Probing cannot close that window; only staying out of the range can. The slot mechanism encoded 49152 as the ephemeral floor, which is the Windows and macOS value, so slot 7 looked like the boundary case while slot 0 was already unsafe on Linux. Windows and macOS were never affected, which is why the flakes were Linux-only. Ports now come from [16384, 32768), below every supported platform's ephemeral range and above the crowded region that holds both ports a developer is likely to be running (6379, 5432, 11211, 8080) and ports this suite itself uses (10000-10002 Azurite). Each test host leases one aligned 32-port block, held by an exclusively locked file for the life of the process, which is what keeps concurrent hosts apart; the OS releases the lock on exit, crash, or kill, so a block frees itself. The lease is taken first and the ports probed second, because software outside the suite, or a server stranded by a killed run, holds ports while holding no lease. Blocks are searched from a start derived from the checkout and the project name, so a project returns to the same ports run after run and netstat output stays interpretable. On collision the search advances by a stride derived from the process, so two hosts that start at the same block diverge rather than walking in lockstep. The stride is forced odd and the block count is a power of two, which together make the walk visit every block exactly once: completing it proves no block is free rather than exhausting a retry budget. The hash is Utility.GetHashCodeWithMix, because the index is cut from the low bits that HashBytes alone leaves unmixed. This replaces slots entirely. Concurrency is no longer capped at seven checkouts, GARNET_TEST_PORT_SLOT is gone and nothing needs setting, and TestPortAssignment, ClusterPortAssignment, and the band-width invariants they required are deleted - projects are identified by assembly name, so adding one needs no central registration. Special cases, each commented where it lives: - Ports 30000-30999 are carved out for test/docker-tests/validate_docker_images.py, which allocates from its own fixed base and can run alongside this suite. TestPortAllocatorTests reads that base out of the Python file so moving either side without the other fails as a test. - The probe binds the IPv4 and IPv6 wildcards separately and exclusively. One dual-mode socket handles IPv4-mapped addresses differently per platform, and a shared bind succeeds alongside a listener on a specific address under that wildcard. - A failed probe deliberately ignores the socket error code: Windows reports AccessDenied for ports excluded by Hyper-V, WSL, or Docker, not AddressAlreadyInUse. - Cluster nodes keep contiguous ports because node n binds base + n; block alignment keeps that run from straddling a boundary. - GARNET_TEST_PORT_BASE pins a base port for reproduction or a fixed firewall opening. Pinned ports are still leased and probed. GarnetServerTcp.Start now reports the endpoint and the operating system's reason when a bind fails, which the raw SocketException names neither of. TestUtils.OnTearDown adds the allocation context, so a conflict reads as software outside the suite rather than as a defect in Garnet or in the test. * Drop stale port-slot instructions from the add-garnet-command skill The skill still told readers to set GARNET_TEST_PORT_SLOT, which no longer exists; test hosts now claim their ports without configuration. * Cover the port allocator's failure paths Reserve's error paths decide whether a run stops clearly or proceeds on ports it does not own, and none of them ran in an ordinary suite: a block another host holds, a pinned base port, and every block exhausted. A regression in any of them would surface as the silent cross-checkout corruption the allocator exists to prevent. Leases in these tests are redirected to a private directory. Holding blocks in the shared one would stop concurrent runs in other checkouts from starting, which is the very failure under test. Writing them found a real defect in the first draft: two tests took the same port from a deterministic helper, and because a reservation holds its lease for the life of the process while binding nothing, the second saw ports that read as free inside a block that was already leased. The helper now checks both, which is the same distinction the allocator draws. BuildPortConflictReport is split out of ReportPortConflictIfBindFailed so the decision and the wording can be asserted without making a real test fail. * Name the alternate port explicitly in ConnectionLimitAcrossListenersTests The fixture built its second listener from TestUtils.TestPort + 1. That is the port TestUtils.AlternateTestPort returns, and it is inside the block this test host leases, so the behavior is unchanged - but the arithmetic only works because the reservation covers two ports, which the expression does not say. Naming the property states the dependency the fixture actually has. * Fix overlapping pinned reservations and misleading bind diagnostics Review of the allocator found four defects, all in paths that only run when something has already gone wrong. ReservePinned leased only the block containing its first port. A pinned base is deliberately not block-aligned, so a run can cross into the next block: base 16415 with two ports leases block 0 while port 16416 sits unleased in block 1. Another host could lease that block and be handed an overlapping run, both probes passing because neither side had bound yet. Every intersected block is now leased. IntersectedBlocks also skips ports outside the allocatable range, where PortBlock is not meaningful - it returned -480 for port 1024 and 738 for port 40000, neither of which names a block. The teardown report fired for every bind failure but always diagnosed a port conflict. GarnetServerTcp also emits "Could not bind" for AccessDenied, which is a platform-reserved range, and AddressNotAvailable, which is an address absent from the machine; neither is another process taking the port. The report now parses the socket error and stays silent unless it is AddressAlreadyInUse. The report read the standalone port field, which ClusterTestContext.ReservePorts never sets, so every cluster bind failure claimed the host had reserved no ports while it held an eight-port run. The allocator now records each reservation and the report looks up the run covering the failing port, which also lets it name the project that holds it. The report asserted that a reserved port cannot have been taken by the OS. That is true only inside the allocatable range. GARNET_TEST_PORT_BASE can pin a run above it, where the kernel really can assign the port to an outbound connection between the probe and the bind - the one collision pinning opts into. Pinned runs outside the range are now described as exactly that, and an endpoint the test built itself is reported as carrying no allocator guarantee at all. Each fix is covered by a test verified by reverting the fix: restoring the single-block lease reports the tail of a straddling run as still available, and removing the error-code gate reports an AccessDenied failure as software outside the suite. | 27 天前 | |
Allocate test ports dynamically from a non-ephemeral range (#2193) * Refuse a second bind on an occupied port on Windows, and add an avalanched hash On Windows GarnetServerTcp set SO_REUSEADDR unconditionally, and cleared SO_REUSEPORT only on Linux, macOS, and FreeBSD. Windows SO_REUSEADDR is permissive: it lets a socket bind an address and port another socket already holds. Two live servers therefore both bound the same port and the OS split incoming connections between them, so a port conflict surfaced as plausible but wrong results rather than as a bind failure. Windows now sets SO_EXCLUSIVEADDRUSE, which refuses the second bind; the Unix paths are unchanged. GarnetServerTcpTests covered this but was skipped on Windows because the behavior it asserts did not hold there. It now runs on every platform, and its TIME_WAIT rebind case confirms exclusive use does not block restarts. Utility.GetHashCodeWithMix applies a MurmurHash3 fmix64 finalizer to HashBytes. HashBytes is a multiply-accumulate, and multiplication propagates information only toward higher bits, so its low bits carry almost no entropy: flipping one input bit changes some output bits with probability 1.000 rather than 0.5. The finalizer's right shifts move entropy back down, which makes it safe to reduce the result modulo a power of two. UtilityTests asserts the avalanche property directly, since that, rather than the hash value, is why the method exists. * Allocate test ports dynamically from a non-ephemeral range Test ports were fixed per sub-project from a base of 33278, shifted by an optional GARNET_TEST_PORT_SLOT offset. That base predates the slot mechanism and sits inside Linux's default outbound-connection range (32768-60999, net.ipv4.ip_local_port_range), so on Linux the kernel could assign a test's port to a connect() between the moment the port was observed free and the moment the server bound it. Probing cannot close that window; only staying out of the range can. The slot mechanism encoded 49152 as the ephemeral floor, which is the Windows and macOS value, so slot 7 looked like the boundary case while slot 0 was already unsafe on Linux. Windows and macOS were never affected, which is why the flakes were Linux-only. Ports now come from [16384, 32768), below every supported platform's ephemeral range and above the crowded region that holds both ports a developer is likely to be running (6379, 5432, 11211, 8080) and ports this suite itself uses (10000-10002 Azurite). Each test host leases one aligned 32-port block, held by an exclusively locked file for the life of the process, which is what keeps concurrent hosts apart; the OS releases the lock on exit, crash, or kill, so a block frees itself. The lease is taken first and the ports probed second, because software outside the suite, or a server stranded by a killed run, holds ports while holding no lease. Blocks are searched from a start derived from the checkout and the project name, so a project returns to the same ports run after run and netstat output stays interpretable. On collision the search advances by a stride derived from the process, so two hosts that start at the same block diverge rather than walking in lockstep. The stride is forced odd and the block count is a power of two, which together make the walk visit every block exactly once: completing it proves no block is free rather than exhausting a retry budget. The hash is Utility.GetHashCodeWithMix, because the index is cut from the low bits that HashBytes alone leaves unmixed. This replaces slots entirely. Concurrency is no longer capped at seven checkouts, GARNET_TEST_PORT_SLOT is gone and nothing needs setting, and TestPortAssignment, ClusterPortAssignment, and the band-width invariants they required are deleted - projects are identified by assembly name, so adding one needs no central registration. Special cases, each commented where it lives: - Ports 30000-30999 are carved out for test/docker-tests/validate_docker_images.py, which allocates from its own fixed base and can run alongside this suite. TestPortAllocatorTests reads that base out of the Python file so moving either side without the other fails as a test. - The probe binds the IPv4 and IPv6 wildcards separately and exclusively. One dual-mode socket handles IPv4-mapped addresses differently per platform, and a shared bind succeeds alongside a listener on a specific address under that wildcard. - A failed probe deliberately ignores the socket error code: Windows reports AccessDenied for ports excluded by Hyper-V, WSL, or Docker, not AddressAlreadyInUse. - Cluster nodes keep contiguous ports because node n binds base + n; block alignment keeps that run from straddling a boundary. - GARNET_TEST_PORT_BASE pins a base port for reproduction or a fixed firewall opening. Pinned ports are still leased and probed. GarnetServerTcp.Start now reports the endpoint and the operating system's reason when a bind fails, which the raw SocketException names neither of. TestUtils.OnTearDown adds the allocation context, so a conflict reads as software outside the suite rather than as a defect in Garnet or in the test. * Drop stale port-slot instructions from the add-garnet-command skill The skill still told readers to set GARNET_TEST_PORT_SLOT, which no longer exists; test hosts now claim their ports without configuration. * Cover the port allocator's failure paths Reserve's error paths decide whether a run stops clearly or proceeds on ports it does not own, and none of them ran in an ordinary suite: a block another host holds, a pinned base port, and every block exhausted. A regression in any of them would surface as the silent cross-checkout corruption the allocator exists to prevent. Leases in these tests are redirected to a private directory. Holding blocks in the shared one would stop concurrent runs in other checkouts from starting, which is the very failure under test. Writing them found a real defect in the first draft: two tests took the same port from a deterministic helper, and because a reservation holds its lease for the life of the process while binding nothing, the second saw ports that read as free inside a block that was already leased. The helper now checks both, which is the same distinction the allocator draws. BuildPortConflictReport is split out of ReportPortConflictIfBindFailed so the decision and the wording can be asserted without making a real test fail. * Name the alternate port explicitly in ConnectionLimitAcrossListenersTests The fixture built its second listener from TestUtils.TestPort + 1. That is the port TestUtils.AlternateTestPort returns, and it is inside the block this test host leases, so the behavior is unchanged - but the arithmetic only works because the reservation covers two ports, which the expression does not say. Naming the property states the dependency the fixture actually has. * Fix overlapping pinned reservations and misleading bind diagnostics Review of the allocator found four defects, all in paths that only run when something has already gone wrong. ReservePinned leased only the block containing its first port. A pinned base is deliberately not block-aligned, so a run can cross into the next block: base 16415 with two ports leases block 0 while port 16416 sits unleased in block 1. Another host could lease that block and be handed an overlapping run, both probes passing because neither side had bound yet. Every intersected block is now leased. IntersectedBlocks also skips ports outside the allocatable range, where PortBlock is not meaningful - it returned -480 for port 1024 and 738 for port 40000, neither of which names a block. The teardown report fired for every bind failure but always diagnosed a port conflict. GarnetServerTcp also emits "Could not bind" for AccessDenied, which is a platform-reserved range, and AddressNotAvailable, which is an address absent from the machine; neither is another process taking the port. The report now parses the socket error and stays silent unless it is AddressAlreadyInUse. The report read the standalone port field, which ClusterTestContext.ReservePorts never sets, so every cluster bind failure claimed the host had reserved no ports while it held an eight-port run. The allocator now records each reservation and the report looks up the run covering the failing port, which also lets it name the project that holds it. The report asserted that a reserved port cannot have been taken by the OS. That is true only inside the allocatable range. GARNET_TEST_PORT_BASE can pin a run above it, where the kernel really can assign the port to an outbound connection between the probe and the bind - the one collision pinning opts into. Pinned runs outside the range are now described as exactly that, and an endpoint the test built itself is reported as carrying no allocator guarantee at all. Each fix is covered by a test verified by reverting the fix: restoring the single-block lease reports the tail of a straddling run as still available, and removing the error-code gate reports an AccessDenied failure as software outside the suite. | 27 天前 | |
Prevent AOF writes during diskless checkpoint streaming (#2252) * Avoid AOF writes during diskless checkpoint replay Apply streamed checkpoint records through the AOF processor replay session, which shares the store but disables AOF recording. Assert the replica AOF tail is unchanged while a checkpoint record is applied. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788da8c4-9305-4866-8d3b-060d62113a0f * Test Vector Set checkpoint imports do not write AOF Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788da8c4-9305-4866-8d3b-060d62113a0f * Assert Vector Set record types across replication, migration and recovery tests Port the record-type assertions from #111 so tests detect Vector Set indexes stored with the wrong record type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788da8c4-9305-4866-8d3b-060d62113a0f * Rename SetCheckpointRecord to ApplySnapshotRecord Diskless sync streams a snapshot, not a checkpoint, to the replica. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788da8c4-9305-4866-8d3b-060d62113a0f * Skip diskless AOF regression tests at runtime outside Debug Use the existing exception-injection ignore helpers instead of compiling the tests out with #if DEBUG. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788da8c4-9305-4866-8d3b-060d62113a0f * Drop redundant DEBUG guard around snapshot pause point Pause points can only be enabled in Debug, so ResetAndWait is a no-op in Release. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788da8c4-9305-4866-8d3b-060d62113a0f * Simplify diskless AOF regression tests Use MeetAndAssignSlots for cluster setup, a shared GetAofTailAddress helper, and a new disposable SuspendParkingScope so the pause/parking handshake no longer needs a flag and manual try/finally. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788da8c4-9305-4866-8d3b-060d62113a0f * Check AOF tail after the entire diskless snapshot Pause full-sync recovery before it resets the AOF addresses, so both regression tests compare the replica tail before and after the complete snapshot transfer rather than after one record. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788da8c4-9305-4866-8d3b-060d62113a0f * Keep Vector Set record-type checks in shared validation helpers Remove scattered TYPE assertions from individual migration and recovery tests while retaining the record-type checks in VectorSetReplicationTestBase validation helpers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788da8c4-9305-4866-8d3b-060d62113a0f * Simplify diskless recovery test synchronization and type checks Apply the recovery pause to full and partial sync. Release parked recovery explicitly within EnabledScope instead of globally suspending parking, and remove the unused suspension scope. Use TYPE in the shared Vector Set validation helper and remove the remaining standalone ad-hoc type assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788da8c4-9305-4866-8d3b-060d62113a0f * Remove duplicate Vector Set AOF test and explain recovery resume Drop VectorSetDisklessCheckpointDoesNotWriteToAof and retain the checkpoint-tail invariant test. Explain why re-enabling the injection flag resumes ResetAndWait rather than injecting another failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788da8c4-9305-4866-8d3b-060d62113a0f * Use the fixture timeout for diskless snapshot arrival Replace the independent 30-second pause-arrival deadline with the existing 15-second fixture timeout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788da8c4-9305-4866-8d3b-060d62113a0f --------- Co-authored-by: Tiago Napoli <tiagonapoli@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 788da8c4-9305-4866-8d3b-060d62113a0f | 14 天前 | |
Allocate test ports dynamically from a non-ephemeral range (#2193) * Refuse a second bind on an occupied port on Windows, and add an avalanched hash On Windows GarnetServerTcp set SO_REUSEADDR unconditionally, and cleared SO_REUSEPORT only on Linux, macOS, and FreeBSD. Windows SO_REUSEADDR is permissive: it lets a socket bind an address and port another socket already holds. Two live servers therefore both bound the same port and the OS split incoming connections between them, so a port conflict surfaced as plausible but wrong results rather than as a bind failure. Windows now sets SO_EXCLUSIVEADDRUSE, which refuses the second bind; the Unix paths are unchanged. GarnetServerTcpTests covered this but was skipped on Windows because the behavior it asserts did not hold there. It now runs on every platform, and its TIME_WAIT rebind case confirms exclusive use does not block restarts. Utility.GetHashCodeWithMix applies a MurmurHash3 fmix64 finalizer to HashBytes. HashBytes is a multiply-accumulate, and multiplication propagates information only toward higher bits, so its low bits carry almost no entropy: flipping one input bit changes some output bits with probability 1.000 rather than 0.5. The finalizer's right shifts move entropy back down, which makes it safe to reduce the result modulo a power of two. UtilityTests asserts the avalanche property directly, since that, rather than the hash value, is why the method exists. * Allocate test ports dynamically from a non-ephemeral range Test ports were fixed per sub-project from a base of 33278, shifted by an optional GARNET_TEST_PORT_SLOT offset. That base predates the slot mechanism and sits inside Linux's default outbound-connection range (32768-60999, net.ipv4.ip_local_port_range), so on Linux the kernel could assign a test's port to a connect() between the moment the port was observed free and the moment the server bound it. Probing cannot close that window; only staying out of the range can. The slot mechanism encoded 49152 as the ephemeral floor, which is the Windows and macOS value, so slot 7 looked like the boundary case while slot 0 was already unsafe on Linux. Windows and macOS were never affected, which is why the flakes were Linux-only. Ports now come from [16384, 32768), below every supported platform's ephemeral range and above the crowded region that holds both ports a developer is likely to be running (6379, 5432, 11211, 8080) and ports this suite itself uses (10000-10002 Azurite). Each test host leases one aligned 32-port block, held by an exclusively locked file for the life of the process, which is what keeps concurrent hosts apart; the OS releases the lock on exit, crash, or kill, so a block frees itself. The lease is taken first and the ports probed second, because software outside the suite, or a server stranded by a killed run, holds ports while holding no lease. Blocks are searched from a start derived from the checkout and the project name, so a project returns to the same ports run after run and netstat output stays interpretable. On collision the search advances by a stride derived from the process, so two hosts that start at the same block diverge rather than walking in lockstep. The stride is forced odd and the block count is a power of two, which together make the walk visit every block exactly once: completing it proves no block is free rather than exhausting a retry budget. The hash is Utility.GetHashCodeWithMix, because the index is cut from the low bits that HashBytes alone leaves unmixed. This replaces slots entirely. Concurrency is no longer capped at seven checkouts, GARNET_TEST_PORT_SLOT is gone and nothing needs setting, and TestPortAssignment, ClusterPortAssignment, and the band-width invariants they required are deleted - projects are identified by assembly name, so adding one needs no central registration. Special cases, each commented where it lives: - Ports 30000-30999 are carved out for test/docker-tests/validate_docker_images.py, which allocates from its own fixed base and can run alongside this suite. TestPortAllocatorTests reads that base out of the Python file so moving either side without the other fails as a test. - The probe binds the IPv4 and IPv6 wildcards separately and exclusively. One dual-mode socket handles IPv4-mapped addresses differently per platform, and a shared bind succeeds alongside a listener on a specific address under that wildcard. - A failed probe deliberately ignores the socket error code: Windows reports AccessDenied for ports excluded by Hyper-V, WSL, or Docker, not AddressAlreadyInUse. - Cluster nodes keep contiguous ports because node n binds base + n; block alignment keeps that run from straddling a boundary. - GARNET_TEST_PORT_BASE pins a base port for reproduction or a fixed firewall opening. Pinned ports are still leased and probed. GarnetServerTcp.Start now reports the endpoint and the operating system's reason when a bind fails, which the raw SocketException names neither of. TestUtils.OnTearDown adds the allocation context, so a conflict reads as software outside the suite rather than as a defect in Garnet or in the test. * Drop stale port-slot instructions from the add-garnet-command skill The skill still told readers to set GARNET_TEST_PORT_SLOT, which no longer exists; test hosts now claim their ports without configuration. * Cover the port allocator's failure paths Reserve's error paths decide whether a run stops clearly or proceeds on ports it does not own, and none of them ran in an ordinary suite: a block another host holds, a pinned base port, and every block exhausted. A regression in any of them would surface as the silent cross-checkout corruption the allocator exists to prevent. Leases in these tests are redirected to a private directory. Holding blocks in the shared one would stop concurrent runs in other checkouts from starting, which is the very failure under test. Writing them found a real defect in the first draft: two tests took the same port from a deterministic helper, and because a reservation holds its lease for the life of the process while binding nothing, the second saw ports that read as free inside a block that was already leased. The helper now checks both, which is the same distinction the allocator draws. BuildPortConflictReport is split out of ReportPortConflictIfBindFailed so the decision and the wording can be asserted without making a real test fail. * Name the alternate port explicitly in ConnectionLimitAcrossListenersTests The fixture built its second listener from TestUtils.TestPort + 1. That is the port TestUtils.AlternateTestPort returns, and it is inside the block this test host leases, so the behavior is unchanged - but the arithmetic only works because the reservation covers two ports, which the expression does not say. Naming the property states the dependency the fixture actually has. * Fix overlapping pinned reservations and misleading bind diagnostics Review of the allocator found four defects, all in paths that only run when something has already gone wrong. ReservePinned leased only the block containing its first port. A pinned base is deliberately not block-aligned, so a run can cross into the next block: base 16415 with two ports leases block 0 while port 16416 sits unleased in block 1. Another host could lease that block and be handed an overlapping run, both probes passing because neither side had bound yet. Every intersected block is now leased. IntersectedBlocks also skips ports outside the allocatable range, where PortBlock is not meaningful - it returned -480 for port 1024 and 738 for port 40000, neither of which names a block. The teardown report fired for every bind failure but always diagnosed a port conflict. GarnetServerTcp also emits "Could not bind" for AccessDenied, which is a platform-reserved range, and AddressNotAvailable, which is an address absent from the machine; neither is another process taking the port. The report now parses the socket error and stays silent unless it is AddressAlreadyInUse. The report read the standalone port field, which ClusterTestContext.ReservePorts never sets, so every cluster bind failure claimed the host had reserved no ports while it held an eight-port run. The allocator now records each reservation and the report looks up the run covering the failing port, which also lets it name the project that holds it. The report asserted that a reserved port cannot have been taken by the OS. That is true only inside the allocatable range. GARNET_TEST_PORT_BASE can pin a run above it, where the kernel really can assign the port to an outbound connection between the probe and the bind - the one collision pinning opts into. Pinned runs outside the range are now described as exactly that, and an endpoint the test built itself is reported as carrying no allocator guarantee at all. Each fix is covered by a test verified by reverting the fix: restoring the single-block lease reports the tail of a straddling run as still available, and removing the error-code gate reports an AccessDenied failure as software outside the suite. | 27 天前 | |
Allocate test ports dynamically from a non-ephemeral range (#2193) * Refuse a second bind on an occupied port on Windows, and add an avalanched hash On Windows GarnetServerTcp set SO_REUSEADDR unconditionally, and cleared SO_REUSEPORT only on Linux, macOS, and FreeBSD. Windows SO_REUSEADDR is permissive: it lets a socket bind an address and port another socket already holds. Two live servers therefore both bound the same port and the OS split incoming connections between them, so a port conflict surfaced as plausible but wrong results rather than as a bind failure. Windows now sets SO_EXCLUSIVEADDRUSE, which refuses the second bind; the Unix paths are unchanged. GarnetServerTcpTests covered this but was skipped on Windows because the behavior it asserts did not hold there. It now runs on every platform, and its TIME_WAIT rebind case confirms exclusive use does not block restarts. Utility.GetHashCodeWithMix applies a MurmurHash3 fmix64 finalizer to HashBytes. HashBytes is a multiply-accumulate, and multiplication propagates information only toward higher bits, so its low bits carry almost no entropy: flipping one input bit changes some output bits with probability 1.000 rather than 0.5. The finalizer's right shifts move entropy back down, which makes it safe to reduce the result modulo a power of two. UtilityTests asserts the avalanche property directly, since that, rather than the hash value, is why the method exists. * Allocate test ports dynamically from a non-ephemeral range Test ports were fixed per sub-project from a base of 33278, shifted by an optional GARNET_TEST_PORT_SLOT offset. That base predates the slot mechanism and sits inside Linux's default outbound-connection range (32768-60999, net.ipv4.ip_local_port_range), so on Linux the kernel could assign a test's port to a connect() between the moment the port was observed free and the moment the server bound it. Probing cannot close that window; only staying out of the range can. The slot mechanism encoded 49152 as the ephemeral floor, which is the Windows and macOS value, so slot 7 looked like the boundary case while slot 0 was already unsafe on Linux. Windows and macOS were never affected, which is why the flakes were Linux-only. Ports now come from [16384, 32768), below every supported platform's ephemeral range and above the crowded region that holds both ports a developer is likely to be running (6379, 5432, 11211, 8080) and ports this suite itself uses (10000-10002 Azurite). Each test host leases one aligned 32-port block, held by an exclusively locked file for the life of the process, which is what keeps concurrent hosts apart; the OS releases the lock on exit, crash, or kill, so a block frees itself. The lease is taken first and the ports probed second, because software outside the suite, or a server stranded by a killed run, holds ports while holding no lease. Blocks are searched from a start derived from the checkout and the project name, so a project returns to the same ports run after run and netstat output stays interpretable. On collision the search advances by a stride derived from the process, so two hosts that start at the same block diverge rather than walking in lockstep. The stride is forced odd and the block count is a power of two, which together make the walk visit every block exactly once: completing it proves no block is free rather than exhausting a retry budget. The hash is Utility.GetHashCodeWithMix, because the index is cut from the low bits that HashBytes alone leaves unmixed. This replaces slots entirely. Concurrency is no longer capped at seven checkouts, GARNET_TEST_PORT_SLOT is gone and nothing needs setting, and TestPortAssignment, ClusterPortAssignment, and the band-width invariants they required are deleted - projects are identified by assembly name, so adding one needs no central registration. Special cases, each commented where it lives: - Ports 30000-30999 are carved out for test/docker-tests/validate_docker_images.py, which allocates from its own fixed base and can run alongside this suite. TestPortAllocatorTests reads that base out of the Python file so moving either side without the other fails as a test. - The probe binds the IPv4 and IPv6 wildcards separately and exclusively. One dual-mode socket handles IPv4-mapped addresses differently per platform, and a shared bind succeeds alongside a listener on a specific address under that wildcard. - A failed probe deliberately ignores the socket error code: Windows reports AccessDenied for ports excluded by Hyper-V, WSL, or Docker, not AddressAlreadyInUse. - Cluster nodes keep contiguous ports because node n binds base + n; block alignment keeps that run from straddling a boundary. - GARNET_TEST_PORT_BASE pins a base port for reproduction or a fixed firewall opening. Pinned ports are still leased and probed. GarnetServerTcp.Start now reports the endpoint and the operating system's reason when a bind fails, which the raw SocketException names neither of. TestUtils.OnTearDown adds the allocation context, so a conflict reads as software outside the suite rather than as a defect in Garnet or in the test. * Drop stale port-slot instructions from the add-garnet-command skill The skill still told readers to set GARNET_TEST_PORT_SLOT, which no longer exists; test hosts now claim their ports without configuration. * Cover the port allocator's failure paths Reserve's error paths decide whether a run stops clearly or proceeds on ports it does not own, and none of them ran in an ordinary suite: a block another host holds, a pinned base port, and every block exhausted. A regression in any of them would surface as the silent cross-checkout corruption the allocator exists to prevent. Leases in these tests are redirected to a private directory. Holding blocks in the shared one would stop concurrent runs in other checkouts from starting, which is the very failure under test. Writing them found a real defect in the first draft: two tests took the same port from a deterministic helper, and because a reservation holds its lease for the life of the process while binding nothing, the second saw ports that read as free inside a block that was already leased. The helper now checks both, which is the same distinction the allocator draws. BuildPortConflictReport is split out of ReportPortConflictIfBindFailed so the decision and the wording can be asserted without making a real test fail. * Name the alternate port explicitly in ConnectionLimitAcrossListenersTests The fixture built its second listener from TestUtils.TestPort + 1. That is the port TestUtils.AlternateTestPort returns, and it is inside the block this test host leases, so the behavior is unchanged - but the arithmetic only works because the reservation covers two ports, which the expression does not say. Naming the property states the dependency the fixture actually has. * Fix overlapping pinned reservations and misleading bind diagnostics Review of the allocator found four defects, all in paths that only run when something has already gone wrong. ReservePinned leased only the block containing its first port. A pinned base is deliberately not block-aligned, so a run can cross into the next block: base 16415 with two ports leases block 0 while port 16416 sits unleased in block 1. Another host could lease that block and be handed an overlapping run, both probes passing because neither side had bound yet. Every intersected block is now leased. IntersectedBlocks also skips ports outside the allocatable range, where PortBlock is not meaningful - it returned -480 for port 1024 and 738 for port 40000, neither of which names a block. The teardown report fired for every bind failure but always diagnosed a port conflict. GarnetServerTcp also emits "Could not bind" for AccessDenied, which is a platform-reserved range, and AddressNotAvailable, which is an address absent from the machine; neither is another process taking the port. The report now parses the socket error and stays silent unless it is AddressAlreadyInUse. The report read the standalone port field, which ClusterTestContext.ReservePorts never sets, so every cluster bind failure claimed the host had reserved no ports while it held an eight-port run. The allocator now records each reservation and the report looks up the run covering the failing port, which also lets it name the project that holds it. The report asserted that a reserved port cannot have been taken by the OS. That is true only inside the allocatable range. GARNET_TEST_PORT_BASE can pin a run above it, where the kernel really can assign the port to an outbound connection between the probe and the bind - the one collision pinning opts into. Pinned runs outside the range are now described as exactly that, and an endpoint the test built itself is reported as carrying no allocator guarantee at all. Each fix is covered by a test verified by reverting the fix: restoring the single-block lease reports the tail of a straddling run as still available, and removing the error-code gate reports an AccessDenied failure as software outside the suite. | 27 天前 | |
Add vector set imports with XVCREATE and XVIMPORT (#2190) * Add vector set imports with XVCREATE and XVIMPORT * formatting * fix VINFO w.r.t. resp3 tests; switch importing indicator to an int * 'fix' interrupted delete test; generally these need some additions, for now XVIMPORT is weird enough we can leave it as a TODO * move XVIMPORT to slow command switch * address nit * rework VXIMPORT to use the quantizer tasks; introduces pauses needed when XVIMPORT is finalizing; remove a bunch of tests and exception injections that aren't quite right given the paradigm shift * formatting * address a bunch of nits * remove test; weird to write directly to store here, needs to arise through the actual request path * rework XVCREATE to use common creation infrastructure; fix tests based on new logic * formatting * exclusive isn't needed for XVIMPORT, it can proceed as shared * remove new lock that would synchronize replication; remove exception smuggling during replication failure, just log; remove test that directly manipulate store to test that smuggling * change termType to an enum, remove duplicate validation * cleanup validation * fix validation errors * align error messages * first whack at removing new GarnetStatus; testcert failures blocking testing at the moment * formatting * fix ReadVectorIndex's handling of VINFO * Add configurable start point for imports * formatting * bump diskann-garnet to 7.0.0 * fixes around index recreation; some tests were using a check incorrectly as well, so de-slop'd that * don't know why there's a rename test in here? also empty keys are legal * restore parallelization of imports post-merge * fix roundRobinReplays handling around replication * bump to diskann-garnet 8.0.0 * slop, this shouldn't be testing finalization --------- Co-authored-by: Jack Moffitt <jackmoffitt@microsoft.com> Co-authored-by: Kevin Montrose <kmontrose@microsoft.com> | 13 天前 | |
[Tests] Start the fuzzy region checkpoints only once every transaction writer is committing (#2211) * Start the fuzzy region checkpoints only once every transaction writer is committing The checkpoint loop is over in about four seconds - it is mostly WaitUntilNextSecond waiting out the one second resolution of LASTSAVE - while each writer first has to establish a multiplexer, a blocking call over async plumbing that four threads were making at once. A writer still connecting when the last checkpoint finished committed nothing, and because AbortOnConnectFail is false and the loop tests cancellation before its first commit, it then exited cleanly: the task completed, WaitAll stayed silent, and the only symptom was a zero count reported as 'no transactions were committed'. The checkpoints now wait for every writer to commit, which both removes the race and makes each checkpoint overlap live traffic rather than an idle primary. A writer that cannot commit faults that wait, so the cause is reported instead of an absent increment; the wait has its own timeout and message; writers are joined on every path, since a multiplexer left alive reconnects to whichever server next binds the port; and SyncTimeout is raised from the five second default, which a transaction of several hundred commands issued during a checkpoint can exceed. * Refactor exception handling to use AggregateException Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> | 21 天前 | |
Fixes for Range Index and Vector Set migration and import validation (#2283) * reject manually migrated Vector Set and Range Indexes log records; validate that migrated vector set indexes are in the correct slot * reject Vector Set index records whose parameters would not pass VADD * tests for forged migration log records * remove pointer from public surface * fix tests * formatting * enforce the same id/value length requirements on the Garnet side as are in the DiskANN side of import_term * dropping indexes on Vector Sets which are in an importing state needs locking; bit of cleanup in task and channel management; fix tests for RI ACLs * address feedback * address feedback | 6 天前 | |
[Cluster] Fix stale offsets and write admission during primary failover (#2253) * fix stale replication offset bug for FAILOVER TO * introduce pause variable * ensure pauseWrites is logical and does not change the node role * fix race when running session misses primary to replica transition at FAILOVER * enforce no checkpoints during failover and ensure that config ref is updated when operations spin on write-pause * refresh snapshot on slot verification retry * ensure that FAILOVER forces writes to spinWait instead of blockin and affecting availability * register background failover task with TaskManager and hook-up timeout to FAILOVER command * mitigate waiter starvation through async await * fix cancellation clearing the pause after demotion * ensure primary tasks are suspended before failover initiates to capture AOF tail * ensure txn safety when flushdb executes while FAILOVER is running * ensure old primary initiates replication stream at FAILOVER success * ensure task cancellation targets only primary tasks for role changing operations * add regression test for REPLICA OF * fix build * ensure commits drain on CommitTask Suspension | 6 天前 | |
Tighten up SetupFixtures and move TestBase to Tsavorite; handle cancellation race condition in maintenance loops (#2162) | 1 个月前 | |
Improvements to test workflow (#1804) | 5 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 27 天前 | ||
| 27 天前 | ||
| 27 天前 | ||
| 27 天前 | ||
| 27 天前 | ||
| 14 天前 | ||
| 27 天前 | ||
| 27 天前 | ||
| 13 天前 | ||
| 21 天前 | ||
| 6 天前 | ||
| 6 天前 | ||
| 1 个月前 | ||
| 5 个月前 |