Upgrading ClickHouse safely: from changelog to rollback

A ClickHouse upgrade fails in one of three ways. The server refuses to start, because something you created years ago no longer exists. The cluster starts but the replicas never converge, or a restart takes long enough that a supervisor kills it. Or everything looks fine and some query now returns a different answer, or a default quietly changes how data is written. The first is the cheapest to find, the last the most expensive, and the order in which this guide works is the order of how cheaply you can catch each one.

This comes from taking replicated clusters across release lines of the 24.x to 25.x kind, then planning the next hop. Queries without a note ran on clickhouse local 26.4; blocks marked “needs a server” touch replication or cluster state and were only syntax-checked with clickhouse format --query. The case for the server helping with all this is in the upgrade guard request, whose five kinds of release-note change I use below.

Choose the target before you choose the date

ClickHouse ships a release every month and marks some of them long-term support. The practical difference is how long a line receives fixes. The supported-versions table lives in SECURITY.md in the repository and the release calendar in the changelog; read both, because the question “is my current line still supported?” is the one most clusters never ask.

In production. A cluster I worked on had been running an LTS line for most of a year. The first thing the pre-upgrade assessment turned up was not in any release note: the line had passed its end of support some weeks earlier. The cluster was also several patch releases behind the final patch of its own line. The question stopped being “should we upgrade” and became “which line do we move to”, with the older LTS line already unsupported and the newest one only weeks old.

The choice was between an older LTS line, many patch releases deep and well documented by other people’s upgrades but buying only about half a year, and the newest line, a few weeks old but buying a full year. If you are chasing a specific fix, check that it exists in the line you pick: here, several memory-related mechanisms aimed at the cluster’s actual failure mode existed only in the newer line, which turned a preference into a measured decision.

Two rules cover most of the rest. Do not skip lines casually: every backward-incompatible change in every release you step over applies at once. And patch first: moving to the last patch of your current line has no incompatible changes by definition and proves your rolling-restart procedure before the real upgrade depends on it.

Turn the changelog into probes

Reading release notes for “anything that might affect us” produces a feeling, not a list. Instead, take the Backward Incompatible Change sections of every release between your version and the target, and for each entry write a probe: a query that returns a number for your estate. An entry with a probe that returns zero is closed; one that returns a row is a work item.

The kinds of change in the request I linked above are a good way to sort them:

1. Changes that make a downgrade impossible (new on-disk format).
2. Changes that break a mixed-version cluster mid-rollout.
3. Changes that alter results with no error.
4. Operational surprises: bigger caches, different thread counts, new background work.
5. Removals.

Removals are where probes pay off immediately, because a removed feature is the one that stops the server from starting. All of these ran on clickhouse local against a small database built from the demo schema plus a JSON column, a token-bloom skip index and a materialized view:

SELECT engine, count() AS tables
FROM system.tables
WHERE database NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA')
GROUP BY engine ORDER BY tables DESC;

┌─engine───────────┬─tables─┐
│ MergeTree        │      3 │
│ Log              │      1 │
│ MaterializedView │      1 │
└──────────────────┴────────┘

Run that first and read the list of engines against the release notes. Then the same for index types, special column types and database engines:

SELECT database, `table`, type FROM system.data_skipping_indices;

SELECT database, `table`, name, type
FROM system.columns
WHERE database NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA')
  AND (type LIKE '%JSON%' OR type LIKE '%Dynamic%' OR type LIKE '%Variant%' OR type LIKE '%Object%');

SELECT name, engine FROM system.databases WHERE engine NOT IN ('Atomic', 'Replicated');

In production. Across a large staging and production estate, the removal probes found a handful of objects, and every one was a throwaway: an abandoned ad-hoc live view, a pair of test tables with an empty ORDER BY on a special engine, and two experimental text indexes in a scratch database. The live view was the hard blocker, because on the target line a server that finds one on disk does not start. The structural part of the upgrade turned out to be a morning of DROP TABLE. The behavioural part took the rest of the planning.

Removals and format changes are probes against metadata. Behaviour changes are probes against text, specifically the SQL stored in your views and materialized views, because that is the part of your application that gets re-parsed by a new server at startup and on every insert.

SELECT database, name, engine
FROM system.tables
WHERE database NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA')
  AND engine IN ('View', 'MaterializedView')
  AND (match(create_table_query, '(?i)\\bneighbor\\s*\\(')
    OR match(create_table_query, '(?i)\\bNOT\\s+[a-z_0-9.()]+\\s+IS\\s+(NOT\\s+)?NULL')
    OR match(create_table_query, '(?i)runningDifference'));

┌─database──┬─name────┬─engine───────────┐
│ analytics │ mv_slow │ MaterializedView │
└───────────┴─────────┴──────────────────┘

The test view was ... WHERE NOT url IS NULL. That pattern matters because the precedence of NOT against IS NULL was changed in a recent release. A new server parses it like this:

SELECT NOT x IS NULL FROM t;   -- clickhouse format prints:
-- SELECT NOT (x IS NULL) FROM t

SELECT NOT 5 IS NULL AS a, (NOT 5) IS NULL AS b, NOT (5 IS NULL) AS c;

┌─a─┬─b─┬─c─┐
│ 1 │ 0 │ 1 │
└───┴───┴───┘

The old parse is the b column. When the old and new parse give different answers for your data, the filter changes silently, and the cluster neither errors nor logs. The only defence is finding the text before the upgrade. A regular expression over create_table_query is crude but sufficient, with two cautions I learned the hard way. Sanity-check each regular expression against a string you know matches and one that must not (col IS NOT NULL must not hit the NOT … IS NULL probe), because a shell-mangled expression returns a confident zero. And run the scan against the deployed objects, not only the repository: the repository was a commit behind in one case I saw, and drift between what is in git and what is in system.tables is exactly what an upgrade exposes.

Deprecated functions fail louder, which is the better outcome, but not at the moment you might expect. On 26.4 I created a view that uses neighbor with allow_deprecated_error_prone_window_functions = 1, restarted, and the view was still listed in system.tables: the metadata loads, so the server starts. The first read without the setting fails:

Code: 721. DB::Exception: Function neighbor is deprecated since its usage is
error-prone (see docs). Please use proper window function or set
`allow_deprecated_error_prone_window_functions` setting to enable it. (DEPRECATED_FUNCTION)

With the setting on, the same SELECT count() over the view returns 100000 rows as before. So a deprecated function is a failure that waits for the first dashboard to hit the view, and the setting is a bridge you can pin while you rewrite. The replacement is a window function; the analyzer tutorial shows the lagInFrame rewrite.

One more probe is worth its cost. Functions and table functions your users actually call are recorded in system.query_log. Rather than guessing which removals matter, ask what ran in the last two weeks (needs a server; system.query_log is empty on a fresh clickhouse local):

SELECT user, arrayJoin(used_table_functions) AS fn, count() AS calls
FROM system.query_log
WHERE type = 'QueryFinish' AND event_date >= today() - 14
GROUP BY ALL ORDER BY calls DESC;

That query pays for itself in the section on grants below.

The compatibility setting and system.settings_changes

Most of the behavioural risk is in default values. ClickHouse records which defaults changed, release by release, in system.settings_changes, and the compatibility setting lets a session or profile use the defaults of an older version. The setting is documented as: use the default settings of a previous version, honouring any setting that has been explicitly changed. Both are worth running on the target binary, because the table only contains history up to its own release. clickhouse local on the new version is the lightest way to get it, with no server involved:

WITH arrayMap(x -> toUInt32(x), splitByChar('.', version)) AS v
SELECT count() AS changes, uniqExact(c.name) AS distinct_settings
FROM system.settings_changes ARRAY JOIN changes AS c
WHERE (v[1], v[2]) > (24, 8) AND (v[1], v[2]) <= (25, 8) AND c.previous_value != c.new_value;

┌─changes─┬─distinct_settings─┐
│     152 │               145 │
└─────────┴───────────────────┘

A year of one release line is a hundred and fifty default flips. Nobody reads that list; you filter it by “settings I have pinned” versus “settings I have not”. The second group is the dangerous one, because a pinned value does not move when the default does. Export your overrides from the old cluster, and join them against the target’s history:

-- needs a server: capture the overrides, per node and per profile
SELECT name, value FROM system.settings WHERE changed ORDER BY name;
SELECT profile_name, setting_name, value FROM system.settings_profile_elements WHERE setting_name != '';

On clickhouse local, with the overrides pasted into a table, the join answers “what does this upgrade do to defaults that I rely on”:

CREATE TABLE my_overrides (name String, value String) ENGINE = Memory;
INSERT INTO my_overrides VALUES ('join_algorithm','hash'), ('max_threads','16'),
  ('query_plan_join_swap_table','false'), ('enable_json_type','0');

SELECT o.name, o.value AS pinned, s.version AS changed_in, s.c.previous_value AS old_default, s.c.new_value AS new_default
FROM my_overrides AS o
INNER JOIN (SELECT version, arrayJoin(changes) AS c FROM system.settings_changes WHERE type = 'Session') AS s
  ON s.c.name = o.name
WHERE s.c.previous_value != s.c.new_value
ORDER BY changed_in;

Pinned rows are flips you are already insulated from; everything else is your review list. A handful of default flips from my run of that table on 26.4 (the version column is where the change landed):

Setting	Changed in	From	To	Why it matters
allow_experimental_analyzer	24.3	0	1	Query analysis path changed; see the analyzer page
parallel_distributed_insert_select	25.7	0	2	INSERT … SELECT over Distributed tables executes differently
output_format_json_quote_64bit_integers	25.8	1	0	JSON clients now receive 64-bit integers as numbers, not strings
replicated_deduplication_window (MergeTree)	25.9	1000	10000	Insert dedup keeps more blocks
replicated_deduplication_window_seconds (MergeTree)	25.10	604800	3600	Dedup memory for retries drops from a week to an hour
async_insert	26.3 (26.2.4 by backport)	0	1	Inserts are buffered server-side by default; 26.2.1 to 26.2.3 still default to 0
deduplicate_blocks_in_dependent_materialized_views	26.2	0	1	Dedup now reaches the targets of materialized views

Read the last four rows twice. The dedup window is a count limit and a time limit on the same set of block hashes, and whichever is hit first governs. The effective window of a table is therefore min(time cap, time to accumulate the count cap), and the crossover is simple arithmetic: with a 10000-block cap and a 3600-second cap, a table that takes more than 10000 inserts an hour is count-limited, and everything slower is time-limited at one hour. Before the flip a table taking a few thousand inserts an hour was count-limited at 1000 blocks, a window of tens of minutes, and the week-long time cap never mattered; after it, the same table is limited to an hour. A retry from a scheduler that fires after the window is no longer deduplicated and writes the data twice. Find your own crossover from the insert rate (needs a server with part_log enabled):

SELECT database, `table`, count() AS inserts_last_hour,
       if(count() > 10000, 'count-limited', 'time-limited (1 h)') AS dedup_window_after
FROM system.part_log
WHERE event_type = 'NewPart' AND event_time > now() - INTERVAL 1 HOUR
GROUP BY ALL ORDER BY inserts_last_hour DESC;

This is a type-3 change, silent and data-affecting, and it is invisible in a rehearsal where nothing retries.

What compatibility does and does not cover

The documentation line sounds like a safety net. Its edges matter, and I checked them on 26.4 rather than trusting the summary. I snapshotted system.settings and system.merge_tree_settings, ran SET compatibility = '25.8', and compared:

CREATE TABLE snap ENGINE = Memory AS SELECT name, value FROM system.settings;
CREATE TABLE snap_mt ENGINE = Memory AS SELECT name, value FROM system.merge_tree_settings;
SET compatibility = '25.8';
SELECT s.name, s.value AS default_26_4, c.value AS compat_25_8
FROM snap AS s INNER JOIN system.settings AS c USING (name)
WHERE s.value != c.value AND s.name IN ('async_insert', 'deduplicate_blocks_in_dependent_materialized_views',
  'deduplicate_insert', 'create_table_empty_primary_key_by_default', 'schema_inference_make_columns_nullable',
  'use_variant_as_common_type', 'input_format_parquet_use_native_reader_v3')
ORDER BY s.name;

┌─name───────────────────────────────────────────────┬─default_26_4─┬─compat_25_8────────────────┐
│ async_insert                                       │ 1            │ 0                          │
│ create_table_empty_primary_key_by_default          │ 1            │ 0                          │
│ deduplicate_blocks_in_dependent_materialized_views │ 1            │ 0                          │
│ deduplicate_insert                                 │ enable       │ backward_compatible_choice │
│ input_format_parquet_use_native_reader_v3          │ 1            │ 0                          │
│ schema_inference_make_columns_nullable             │ 3            │ 1                          │
│ use_variant_as_common_type                         │ 1            │ 0                          │
└────────────────────────────────────────────────────┴──────────────┴────────────────────────────┘

Counting every row, 73 query-level settings moved. The same comparison on system.merge_tree_settings moved zero: replicated_deduplication_window_seconds stayed at the new one-hour default under compatibility = '25.8', which means the dedup-window flip above is not undone by pinning compatibility. You pin it by name, in the MergeTree settings, or you accept it. And system.server_settings is outside it altogether: background_pool_size and the merge-concurrency ratio read the same with and without compatibility, and changed stays 0.

Three more things to hold in your head:

• An explicit value beats compatibility. A setting you name is never touched, which is what makes a pin safe and also what makes it a decision you must remember to revisit.
• Compatibility switches off the new behaviour you upgraded for. A cluster upgraded to a newer line to get a join-spilling fix kept the fix disabled for as long as the pin held, because the setting that enables the fix was one of the compatibility-managed query-level defaults. The pin is a ramp, not a destination; the plan must include the step that relaxes it.
• It must be in place before the first start. The release-note guidance I followed was explicit that you cannot go back unless the setting is already there when the new version starts. Set it while the old binary is still serving, in the profile every user inherits, and in every profile. In one rollout the pin was added to the default profile only. A restricted read-only profile used by the public API did not get it, and would have taken the new defaults on day one. Search every profile that exists, not just the one you use.

<profiles>
    <default>
        <compatibility>25.8</compatibility>
    </default>
    <readonly>
        <compatibility>25.8</compatibility>
    </readonly>
</profiles>

To see the effect on one query without touching a profile, use the setting inline: SELECT … SETTINGS compatibility = '25.8'.

The analyzer is a separate project

The new query analyzer became the default in 24.3, and the switch that turns it off, enable_analyzer, is still honoured on 26.4 (I ran SET enable_analyzer = 0; SELECT 1 to confirm). That is exactly why it is dangerous: a cluster can carry enable_analyzer = 0 in a profile for years, every release note says “still works”, and the day it is removed you are both upgrading and migrating the query engine. Treat the pin as a debt and schedule the move off it independently of any version hop. The rejected queries and their rewrites are in Moving to the new analyzer; the upgrade-specific advice is to replay real queries: run a few days of distinct normalized_query_hash values from system.query_log with the analyzer on in staging and diff the result hashes.

Rehearse on staging, and make staging look like production

A rehearsal catches what probes do not, if staging resembles production in the dimensions that matter. These bit:

Part count, not just data size. Startup time is largely a matter of loading part metadata, so the cold-start time of a node scales with how many parts it holds, and with how slow its disk is. Measure the real thing:

-- ran on clickhouse local; run on every node
SELECT count() AS active_parts, formatReadableSize(sum(bytes_on_disk)) AS on_disk
FROM system.parts WHERE active;

On slow magnetic volumes a node with tens of thousands of active parts needed over ten minutes to become ready. The same count on fast volumes loaded in a few minutes. A liveness probe that gives a process a couple of minutes to answer will kill it repeatedly, and every kill restarts the load, so the node never comes up. If you run under an orchestrator or a supervisor, give startup its own generous health-check budget (separate from steady-state liveness), sized as a multiple of the measured cold load time. Measure it in staging; do not extrapolate from data size.

Configuration from the real deployment path. Render the config exactly as production will receive it and diff it against what is running, rather than reviewing the change you intended. In one rehearsal the intended change was a one-line image bump; the render against the live release showed it would also pick up two unrelated, already-merged changes, because the deployment source had moved since the live release. Both happened to be good. The difference between “what I applied” and “what I thought I applied” was visible only in the render diff.

In production. Starting the new binary on the cluster’s real configuration, in a throwaway container, caught the best surprise of the rehearsal. An unrecognised top-level element in one configuration file was tolerated by the target line and rejected by the line after it, where the server refuses to start. The server logged the right error, but the container exited with status 137, which is also the status a container runtime reports for an out-of-memory kill. For a few minutes it read as an OOM. It is not: 137 is ClickHouse’s own UNKNOWN_ELEMENT_IN_CONFIG code (SELECT errorCodeToName(137)) and the fix was the server setting skip_check_for_incorrect_settings, or removing the stray element. The boot test cost minutes and removed a hard failure from the maintenance window.

Make “boot the exact production configuration under the target binary” a standing step: it catches stricter validation, which the changelog rarely mentions.

Grants and privileges. The query-log probe above exists for this. A new release introduced a privilege on the table-function family that reaches other servers, REMOTE:

SELECT privilege, level, parent_group FROM system.privileges WHERE privilege ILIKE '%remote%';

┌─privilege─┬─level──┬─parent_group─┐
│ REMOTE    │ GLOBAL │ ALL          │
└───────────┴────────┴──────────────┘

A restricted user whose dashboards used clusterAllReplicas() lost the ability to run it after the upgrade, while reads of Distributed tables, which go through a different path, kept working. The fix is a grant (GRANT READ ON REMOTE TO …, on every node for SQL-managed users, or ON CLUSTER), but you find the users who need it in the query-log probe, before the window and not after.

Replay what you can. Insert a copy of recent production data on the new version and diff aggregates against the old. Retries and dedup are exercised only if the rehearsal retries.

Mixed versions, rolling upgrades, and the alternative

Replicas on different versions coexist during a rolling restart, and a fetch carries a part from one replica to another, so the receiver must understand the part’s format. That is why the “breaks a mixed-version cluster” category exists. What keeps a rolling upgrade safe is a gate between nodes, not speed:

1. Upgrade one replica of one shard first (a canary). Hold there.
2. Run the gate below. Do not proceed until every row is clean.
3. Continue one node at a time. Never take two replicas of the same shard down together.
4. Remember that during the roll, initiator and remote nodes of a Distributed query run different code.

-- needs a server: run on the node you just restarted, then on its peers
SELECT version() AS version, uptime() AS uptime_s;

SELECT count() AS replicas_not_ready FROM system.replicas
WHERE is_readonly OR is_session_expired OR queue_size > 20 OR absolute_delay > 60 OR lost_part_count > 0;

SELECT count() AS stuck_mutations FROM system.mutations WHERE NOT is_done AND latest_fail_reason != '';

SELECT count() AS queue_errors FROM system.replication_queue WHERE last_exception != '';

SELECT count() AS detached_parts FROM system.detached_parts;

Keeper itself is a separate ensemble with its own version. In one case the coordination servers ran a newer patch release of the target line than the data servers for weeks before the data servers were upgraded, and no problem appeared, because the two speak a stable wire protocol and newer operations are negotiated through feature flags. Upgrade the two tiers independently and never in the same window unless you have to.

The alternative is to stop the fleet, upgrade, start it. It costs downtime and removes the whole class of mixed-version failure. With ingest paused, the image pre-pulled on every node and a tested cold start, the cutover took well under half an hour. Choose it when the changelog lists a protocol or format change that makes a rolling upgrade questionable; choose rolling when you can pause nothing.

In production. The same procedure was used twice, once on staging and once on production, in a deliberately boring shape. Pause the writers. Bring up one node alone and gate on it: version, number of replicas registered with Keeper and zero read-only, a create–insert–select–drop probe on a replicated table, and an error-log grep for coordination exceptions. Only then bring up the rest. The gate on the lone node passed ahead of its abort checkpoint, and the fleet came up with zero restarts. The abort checkpoints were written down in advance as clock times, with the decision rule: by the second one, extend or roll back, because past it, writes landing on the new version make a rollback dirty.

Downgrade limits: the one-way door

The important fact about a downgrade is that ClickHouse does not guarantee one. A new binary may write parts in a format the old one cannot read, and from the first insert that part exists. You can see how explicit the format is in any current part directory. This is the list for one small compact part on 26.4:

checksums.txt  columns_substreams.txt  columns.txt  count.txt  data.bin  data.cmrk4
default_compression_codec.txt  metadata_version.txt  partition.dat  primary.cidx
serialization.json

and the substreams file begins with a version line:

columns substreams version: 1

That listing does not say which of these an older release would refuse; the point is that parts carry format versions of their own, and the changelog entries that matter for rollback are the first category above. For per-release examples (a nullable-index format, sparse columns by default, per-substream marks in compact parts) see the upgrade guard request, which quotes the release-note wording and warns that a correct downgrade may not be possible.

The safe model is: the upgrade is one-way for every part written after it. Everything about rollback planning follows from that.

• compatibility can narrow the window, not close it. It restores old defaults for new writes only where the format change sits behind a setting. Set it before the new binary’s first start, and treat a late rollback as “rewrite what the new version wrote”, not as “put the old image back”.
• Rollback is cheap only before meaningful writes. Time-box the decision. Within the window, a rollback is a binary swap. After new-format parts land on every replica, it is a restore.
• A backup is your only real undo. In one case no point-in-time backup existed and volume snapshots were consciously skipped; the plan was written around that (a single node first, a hard decision time, a rollback that worked only until traffic resumed). That is legitimate when made on purpose.
• Pre-pull the old image and keep the old configuration on every node. A downgrade that starts with a registry pull on a cluster that is down is a bad hour.

Things the first boot does that no one tells you

• System log tables are renamed and recreated. When a release changes the schema of query_log, text_log, metric_log and friends, the server renames the existing table with a numeric suffix and creates a new one. The old table stays, with all its data, and each upgrade can add another generation. On a busy cluster these orphans reached roughly a terabyte in one table and more across several, found weeks later in a disk alert. Look for them (empty on a fresh install; ran on clickhouse local):

SELECT name, formatReadableSize(total_bytes) AS size
FROM system.tables
WHERE database = 'system' AND match(name, '_[0-9]+$')
ORDER BY total_bytes DESC;

The query log baselines you captured before the upgrade also live in the renamed table; query them there before you drop it.

• Defaults on inserts move. async_insert turning on by default changed when rows became visible for one set of clients and did nothing for another; the second group was INSERT … SELECT and engine-driven ingestion. Async inserts explained describes the trade-off. Know which of your writers is which before the window, not after.

Pre-flight checklist

Run these on the old cluster, a week before and again an hour before. Save the output; you will diff it against the same run after the upgrade.

Check	Where	Pass condition
Current line is supported; target chosen against the fixes you need	SECURITY.md, changelog	Written down with a reason
Removal probes (engines, indexes, column types, database engines)	clickhouse local	Zero rows, or each row has an owner
Stored SQL scan for changed parsing, deprecated and renamed functions	system.tables	Zero hits on deployed objects, not just the repo
Overrides joined against system.settings_changes	clickhouse local	Every unpinned flip reviewed
Pinning decided for query, MergeTree and server settings separately	profile + config	Compatibility set in every profile, MergeTree flips pinned by name
Boot the real configuration on the target binary	scratch container	Starts, and your pins show changed = 1
Grants audit from system.query_log	the query above	Every user of cluster functions mapped
Cold-start time measured (parts, disk)	staging	Health-check budget is a multiple of it
Keeper tier healthy; version plan for it	system.replicas, Keeper commands	Independent of the data tier
Backup or snapshot exists, or the lack of one is a recorded decision	operations	Named owner
Rollback checkpoints written as clock times	runbook	Includes the decide-by time

The health snapshot, all on the old cluster (needs a server; syntax-checked):

SELECT database, `table`, is_readonly, is_session_expired, queue_size, absolute_delay,
       lost_part_count, active_replicas, total_replicas
FROM system.replicas
WHERE is_readonly OR is_session_expired OR queue_size > 20 OR absolute_delay > 60
   OR lost_part_count > 0 OR active_replicas < total_replicas;

SELECT database, `table`, mutation_id, parts_to_do, latest_fail_reason
FROM system.mutations WHERE NOT is_done;

SELECT database, `table`, type, num_tries, last_exception
FROM system.replication_queue WHERE last_exception != '' ORDER BY num_tries DESC LIMIT 20;

SELECT database, `table`, reason, count() AS parts FROM system.detached_parts GROUP BY ALL;

SELECT name, value FROM system.errors WHERE value > 0 ORDER BY value DESC LIMIT 20;

Clean results on the old cluster prove the baseline; a read-only replica that predates the window must not be blamed on the upgrade.

The rollback plan

Write it before the window, as a short list of conditions and actions, and make one person responsible for calling it.

1. Abort before the first write on the new version. If the canary gate fails, put the previous image back on that node. Nothing new has been written and the rest of the fleet is untouched.
2. Hold a decide-by time. Define the clock time after which writes on the new version are significant, and decide extend-or-roll-back at that time, not later.
3. If you must go back after that point, the unit of recovery is the part: restore from backup or re-create the affected tables from a replica that was never upgraded, then re-insert. This is slow, which is the whole argument for the decide-by time.
4. Use the pins first. If the behaviour is the problem rather than the binary, changing a setting back is a faster lever than a downgrade.

After the window, relax the pins one default at a time, a day apart, with the baseline queries as the measurement. Each default that moves from “pinned” to “new” is a small upgrade of its own; relax the one with the largest blast radius last.