An upgrade guard for ClickHouse: defaults that outlive a downgrade

Of all the requests I have filed, #97500 is the longest and was closed the fastest: opened on 2026-02-20 at 16:22 UTC, closed at 00:42 UTC the next day. It is less a feature request than a proposal, so it earns a longer look, and the interesting part is not that it was declined but what the two-line answer does and does not cover.

The design tension

ClickHouse ships new defaults aggressively. That is a feature: users get the faster path without reading a changelog. The cost is that some defaults decide what is written to disk, and once a part exists in the new format an older binary may not be able to read it. From that moment the upgrade is one-way, and nothing in the product tells you which flip it was.

The issue sorted what release notes contain into five kinds of change. The groups are the useful part, so here they are with examples quoted in the issue from the release notes (check the notes themselves before relying on a specific item):

• Downgrade becomes impossible. For the 22.3 nullable-index file format the release note, as the issue quotes it, says that 21.8 “can not read those files, so a correct downgrade may not be possible”. Other items: sparse columns by default (23.8) and per-substream marks in compact parts (25.8).
• Mixed-version clusters break during a rolling upgrade. A replication protocol change for TTL merges (20.10), ALTER formatting with parentheses (25.3).
• Wrong results with no error. The issue’s example is the argument order of the H3 geo functions changing.
• Operational surprises. Larger default caches, a thread-count default that doubles CPU.
• Removals. Index types and argument types that stop existing.

The tension: the project wants fresh installs to get the best defaults immediately, while an operator with a live cluster wants to take a new binary without taking on every new on-disk format at the same moment. Those are two different populations, and a single default serves only one.

What I proposed

Two small mechanisms and a policy, all on the server.

A version history the server keeps. An append-only file in the data directory, one line per start: version, start time, stop time, reason. A start that finds the previous line still marked running knows there was a crash. From it the server knows whether this start is an upgrade, what ran before, what the oldest version to have touched the data is, and whether a downgrade ever happened. Queryable as system.version_history. The cost is a few lines of code at startup and shutdown.

Impact tags on settings. Each risky setting carries a category (format_change, protocol_change, behavioral, resource_change, removal) and a SAFE_SINCE version: the oldest version able to read data written with the setting on.

A configured downgrade_target and a policy. safe holds back every format and protocol change newer than the target, plus behavioral changes; adventurous holds back only format and protocol changes that break the target; yolo is what ClickHouse does now and the proposed default for fresh installs. The server logs what it held back and what it enabled, and lets you release features by moving the target forward as confidence grows. Changes that are not behind a setting (a library upgrade, a removed feature) would live in a registry that lets safe refuse to start with a clear error.

I cited prior art in the issue: CockroachDB’s cluster.preserve_downgrade_option, which holds new features back until the admin finalizes the upgrade; PostgreSQL’s separate, explicit pg_upgrade; and IBM Db2 function levels. The issue also says the idea is mine and the text was written by an AI model; I stand by the idea, and the length is the model’s.

What the maintainers said

Two comments. First SaltTan:

Information about settings' changes is available in `system.settings_changes`.
It is not categorized though.
For safer upgrades it is possible to use the `compatibility` setting

Then alexey-milovidov closed the issue:

We use the `compatibility` setting - it works well.

That is a reasonable decision on the project’s terms. A mechanism for the common case exists, and the proposal would add a subsystem, a metadata macro on every setting declaration, and a registry that has to stay honest for the life of the product. The cost sits on the maintainers; the benefit sits with operators who have not yet needed it. I do not read the answer as saying the problem is imaginary. SaltTan concedes the weak point in the second sentence: the log exists, but “it is not categorized though”.

What the existing answer gives you

Both pieces exist on 26.4 and are worth using, so let me show them. system.settings_changes has three columns: type (Session for query settings, MergeTree for table settings), version, and an array changes of (name, previous_value, new_value, reason):

SELECT type, version, length(changes) AS n
FROM system.settings_changes ORDER BY version DESC, type LIMIT 4;

Session     26.4    17
MergeTree   26.4    4
Session     26.3    29
MergeTree   26.3    12

The oldest entry is from 18.12. Each change can be unpacked, for example to find the release that flipped a default:

SELECT version, tupleElement(c,'name') AS name,
       tupleElement(c,'previous_value') AS prev, tupleElement(c,'new_value') AS new
FROM system.settings_changes ARRAY JOIN changes AS c
WHERE name = 'optimize_syntax_fuse_functions';

26.3   optimize_syntax_fuse_functions   0   1

compatibility pins defaults to an earlier release. The 26.3 flip above goes back when you ask for 26.2:

SELECT getSetting('optimize_syntax_fuse_functions') AS current_default;
SET compatibility = '26.2';
SELECT getSetting('optimize_syntax_fuse_functions') AS under_compat_26_2;

true
false

This is genuinely useful, and it is closer to the guard than my first reading allowed. Set compatibility in a profile at the old version, upgrade the binaries, then move the value forward release by release. That is downgrade_target moving forward, done by hand and with no history file. The semantics in the current settings docs are sensible too: settings you set explicitly are honored, and changes marked Ignore in the history block rollback of that change and all earlier changes to the same setting. The server source also has a MergeTreeSettings::applyCompatibilitySetting, so table-level defaults are meant to follow the same rule.

What it cannot give you

Three gaps remain, and they are exactly the ones the RFC was about.

The log is uncategorized. Between 25.3 and 26.4 the log records 131 session-setting and 29 table-setting default flips (rows where the previous and new value differ). Nothing says which of them touch the on-disk format. You can approximate by searching the reason text, which is what I do. Between 25.3 and 26.4 this query finds table settings whose reason mentions format, serialization, marks or compatibility:

SELECT version, tupleElement(c,'name') AS setting,
       tupleElement(c,'previous_value') AS prev, tupleElement(c,'new_value') AS new
FROM system.settings_changes ARRAY JOIN changes AS c
WHERE type = 'MergeTree'
  AND prev != new
  AND arrayMap(x -> toUInt32(x), splitByChar('.', version)) > [25, 3]
  AND arrayMap(x -> toUInt32(x), splitByChar('.', version)) <= [26, 4]
  AND (tupleElement(c,'reason') ILIKE '%format%' OR tupleElement(c,'reason') ILIKE '%serializ%'
       OR tupleElement(c,'reason') ILIKE '%mark%')
ORDER BY arrayMap(x -> toUInt32(x), splitByChar('.', version)), setting;

25.8   write_marks_for_substreams_in_compact_parts                 0              1
25.11  serialization_info_version                                  basic          with_types
25.11  string_serialization_version                                single_stream  with_size_stream
25.12  dynamic_serialization_version                               v2             v3
25.12  object_serialization_version                                v2             v3
25.12  object_shared_data_serialization_version                    map            advanced
25.12  object_shared_data_serialization_version_for_zero_level_parts  map         map_with_buckets
26.3   propagate_types_serialization_versions_to_nested_types      0              1

The first row is one of the items the issue named, so the heuristic finds it; the 25.11 row has the reason “Change to the newer format allowing custom string serialization”. A keyword search is a heuristic, not a guarantee: it misses anything whose reason is phrased differently, and a human has to decide whether each row can block your downgrade. That is the categorization SaltTan says is missing.

Note the version comparison. Versions are strings, and string order puts 25.10 before 25.2; I compare them as arrays of integers. Skipping this gives silently wrong “newer than” filters on 25.10 to 25.12.

It does not know your data. compatibility is a setting on the current process. Nothing on the server records which versions have written the parts you hold, so the question “can I still go back to 25.3?” has no answer in SQL. The server has no history table; system.version_history and system.upgrade_guard do not exist:

SELECT * FROM system.upgrade_guard;

Code: 60. DB::Exception: Unknown table expression identifier 'system.upgrade_guard' in scope
SELECT * FROM system.upgrade_guard. (UNKNOWN_TABLE)

It does not cover what is not a setting. Library upgrades that change checksums between replicas, removed index types and altered function semantics are not defaults to flip back, so compatibility has nothing to restore.

One more caution from my own run: on 26.4 clickhouse local, after SET compatibility = '25.3', getMergeTreeSetting('write_marks_for_substreams_in_compact_parts') still returned true, and a table created in that session wrote the same part files as one created without it. I did not trace why. Treat “does compatibility pin the on-disk format of new parts” as something to test on your own version before you rely on it.

What it teaches about ClickHouse’s boundaries

The project is a library of fast-moving defaults held together by a compatibility lever, not by a state machine. That works because most changes are safe to roll back and because the maintainers read the same release notes the rest of us do. The guard would have moved some of that knowledge from people into the server, and the maintainers chose not to take on a registry that has to be right forever.

My practice is the cheap version of the RFC. Keep a plain log of versions that have run on each cluster. Write down the oldest version you can still return to. Before any upgrade, diff system.settings_changes between the two versions with the numeric version filter above. Pin compatibility for the first days, then move it forward. If I had to keep one piece of the RFC it would be the first: a server that knows its own history is nearly free, and everything else could be built on it by someone else.