JJakub PliszkaForce migration key for composite chunk boundaries
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Add opt-in ANALYZE TABLE on the ghost table before cut-over Add --analyze-ghost-table-before-cutover. When set, cutOver() runs an explicit ANALYZE TABLE on the ghost table after the postpone gate releases — before atomicCutOver() takes the source lock and before --test-on-replica stops replication — logs the elapsed milliseconds on success, and aborts the migration (fatal) if the ANALYZE fails, rather than swap in a table with stale InnoDB statistics. The abort exits synchronously (Log.Fatale), not via a retriable return — a plain return re-runs cutOver() and the ANALYZE up to --default-retries. Because ANALYZE TABLE reports table-level failures (missing table, storage-engine errors) as Msg_type Error rows in its result set while succeeding at the protocol level, the result rows are inspected and cut-over is refused unless ANALYZE reports status OK with no Error rows; privilege-style failures surface as statement errors on the same abort path. Without this, a freshly swapped table can briefly serve traffic with a near-zero row estimate, which the optimizer may cost as a free full scan on hot query paths, flipping plans until statistics are recomputed. Issue #1418 / PR #1419 propose an ANALYZE for the same reason; this variant corrects two defects there — the ANALYZE runs after the postpone gate (so a postponed cut-over still gets fresh statistics) and a failed ANALYZE aborts instead of being ignored. Opt-in, matching the maintainers' ask on #1419 (ANALYZE cost grows with partition count, and the statement replicates). The result-row inspection is extracted as classifyAnalyzeTableResult, a pure, DB-free function, and covered by: - TestClassifyAnalyzeTableResult: a table test over status-OK, case folding, an error row (alone and alongside a status-OK row), a status-OK row followed by a later error row (rows are scanned fully, not short-circuited), a non-OK status, and an empty result. Each refusal asserts the underlying cause via ErrorContains. - ApplierTestSuite.TestAnalyzeGhostTable (real MySQL): happy path; the fail-open regression (dropping the ghost table makes ANALYZE return an Error row with no statement error, which the row inspection must refuse); and the statement-error branch (a closed connection is refused via the distinct error path). Also fixes a pre-existing suite bug surfaced while adding the test above: testify's suite runner calls TearDownSuite() (capital D), but the applier, migrator, and streamer suites all spelled it TeardownSuite(), so the method never matched the interface and the MySQL testcontainer was never terminated. Renamed in all three suites. Co-authored-by: wangzihuacool <wangzihuacool@163.com> Signed-off-by: Vandhana Selvaprakash <vandhana.selvaprakash@airtable.com> | 1 个月前 | |
fix: clone GTID coordinates for reader and syncer ownership The binlog reader updates its current coordinates while streaming, while go-mysql's BinlogSyncer retains and mutates the GTID set passed to StartSyncGTID. Clone the caller-provided coordinates for the reader and clone the GTID set again for the syncer. This prevents both components from mutating the caller's reconnect coordinates and eliminates concurrent access to the same MysqlGTIDSet. | 29 天前 | |
Add opt-in ANALYZE TABLE on the ghost table before cut-over Add --analyze-ghost-table-before-cutover. When set, cutOver() runs an explicit ANALYZE TABLE on the ghost table after the postpone gate releases — before atomicCutOver() takes the source lock and before --test-on-replica stops replication — logs the elapsed milliseconds on success, and aborts the migration (fatal) if the ANALYZE fails, rather than swap in a table with stale InnoDB statistics. The abort exits synchronously (Log.Fatale), not via a retriable return — a plain return re-runs cutOver() and the ANALYZE up to --default-retries. Because ANALYZE TABLE reports table-level failures (missing table, storage-engine errors) as Msg_type Error rows in its result set while succeeding at the protocol level, the result rows are inspected and cut-over is refused unless ANALYZE reports status OK with no Error rows; privilege-style failures surface as statement errors on the same abort path. Without this, a freshly swapped table can briefly serve traffic with a near-zero row estimate, which the optimizer may cost as a free full scan on hot query paths, flipping plans until statistics are recomputed. Issue #1418 / PR #1419 propose an ANALYZE for the same reason; this variant corrects two defects there — the ANALYZE runs after the postpone gate (so a postponed cut-over still gets fresh statistics) and a failed ANALYZE aborts instead of being ignored. Opt-in, matching the maintainers' ask on #1419 (ANALYZE cost grows with partition count, and the statement replicates). The result-row inspection is extracted as classifyAnalyzeTableResult, a pure, DB-free function, and covered by: - TestClassifyAnalyzeTableResult: a table test over status-OK, case folding, an error row (alone and alongside a status-OK row), a status-OK row followed by a later error row (rows are scanned fully, not short-circuited), a non-OK status, and an empty result. Each refusal asserts the underlying cause via ErrorContains. - ApplierTestSuite.TestAnalyzeGhostTable (real MySQL): happy path; the fail-open regression (dropping the ghost table makes ANALYZE return an Error row with no statement error, which the row inspection must refuse); and the statement-error branch (a closed connection is refused via the distinct error path). Also fixes a pre-existing suite bug surfaced while adding the test above: testify's suite runner calls TearDownSuite() (capital D), but the applier, migrator, and streamer suites all spelled it TeardownSuite(), so the method never matched the interface and the MySQL testcontainer was never terminated. Renamed in all three suites. Co-authored-by: wangzihuacool <wangzihuacool@163.com> Signed-off-by: Vandhana Selvaprakash <vandhana.selvaprakash@airtable.com> | 1 个月前 | |
Force migration key for composite chunk boundaries | 15 天前 | |
StatsD instrumentation: row-copy, backlog, lag, throttle, cut-over, query latency, sleep (#1701) * metrics: introduce unified Emitter interface and consolidate go_runtime helpers Introduce a single Emitter interface (Gauge, Count, Histogram) on metrics.Client so all metric helpers share one testable contract. The interface replaces the narrow MemStatsGaugeEmitter type used only by the Go runtime reporter. - Add Emitter interface and Histogram method on Client. - Move go_runtime.go and go_runtime_test.go into emit.go and emit_test.go so the metrics package has a single place for helper functions and tests. - Switch MigrationContext.Metrics from *metrics.Client to metrics.Emitter so consumers can be tested with spies. * Refactor status reporting around migrationProgressSnapshot. Sample row-copy, DML, backlog, and lag once per tick into a snapshot passed to printStatus, with reportStatus as the single entry point for status output. Co-authored-by: Cursor <cursoragent@cursor.com> * metrics: emit row-copy progress gauges Add gh_ost.row_copy.rows_copied, gh_ost.row_copy.rows_estimate, and gh_ost.dml.events_applied gauges on each reportStatus tick, sampled from migrationProgressSnapshot. * metrics: emit binlog backlog gauges Add gh_ost.binlog.backlog_size, gh_ost.binlog.backlog_capacity, and gh_ost.binlog.backlog_utilization gauges on each reportStatus tick from the applyEventsQueue depth captured in migrationProgressSnapshot. * metrics: emit replication and heartbeat lag gauges Add gh_ost.lag.replication_seconds and gh_ost.lag.heartbeat_seconds gauges on each reportStatus tick, tagged with throttled:true|false. These are point-in-time readings (not distributions), so gauges are used rather than histograms — DogStatsD histogram aggregation exposes count/max series that do not match the log line lag values in Prometheus/Grafana. * metrics: emit throttle active and throttled-interval metrics Record throttle active state at a debounced cadence (gh_ost.throttle.active) and emit duration plus event metrics when a throttled interval completes (gh_ost.throttle.duration_seconds histogram and gh_ost.throttle.events_total count), each tagged with the throttling reason. * metrics: emit cut-over phase, attempt, and total duration metrics Add cut-over metric helpers and instrument cut-over attempts, phase durations, and terminal duration. Metrics emitted: - gh_ost.cut_over.attempts_total tagged with outcome - gh_ost.cut_over.phase_duration_milliseconds tagged with phase and outcome - gh_ost.cut_over.total_duration_milliseconds tagged with outcome Phase coverage includes the magic lock, original table lock, magic rename, and unlock paths. Durations are reported in milliseconds to preserve sub-second granularity. The atomic rename phase duration is recorded after the rename completes so the histogram reflects the full operation. * metrics: emit query latency histogram Emit gh_ost.query.duration_milliseconds for representative source-side and target-side queries (row count and binlog apply), tagged with side, kind, and outcome (ok|error). Helper validates inputs and is nil-safe. * metrics: emit per-stage sleep histograms Add sleep metric helpers and instrument the main migration sleep/wait paths. Metrics emitted: - gh_ost.sleep.duration_milliseconds tagged with stage - gh_ost.sleep.total_milliseconds tagged with stage Stages covered: - cut_over_postpone - chunk_throttle - retry_backoff - replica_wait Use millisecond units so sub-second waits, such as replica polling and nice-ratio throttling, are not truncated to zero. Skip sub-millisecond chunk-throttle samples to avoid emitting zero-valued sleeps that would distort the histogram. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Jan Grodowski <jan.grodowski@shopify.com> | 2 个月前 | |
Fix mixed version topology discovery | 17 天前 | |
Force migration key for composite chunk boundaries | 15 天前 |