Migrating from ZooKeeper to ClickHouse Keeper

Moving a replicated cluster from ZooKeeper to ClickHouse Keeper looks like the least dangerous migration there is: Keeper speaks the ZooKeeper client protocol, a replicated table does not care which answers, and ClickHouse ships a converter for the data. The data is the part that goes wrong. A conversion that exits with status 0 can leave a tree that boots, serves reads, passes a casual comparison, and rejects every insert into your busiest table days later. This page covers the mechanism, the plan that follows from it, and the housekeeping after a successful cut-over.

If you need the model of what a replicated table keeps in the coordination service (the log, the queue, the block hashes, the replica nodes), read How replication works first. Statements that need a coordination service were only syntax-checked with clickhouse format --query on 26.4; the tools were checked against their --help output on that version; the znode comparisons ran on clickhouse local with synthetic listings and are marked.

Why move at all

The reasons are operational, not performance headlines:

• One fewer runtime. Keeper ships in the ClickHouse binary, is written in C++, and has no JVM to size or to pause.
• It uses Raft rather than ZAB, and writes compressed logs and snapshots. A ZooKeeper snapshot file of roughly a hundred megabytes came out at about a sixth of that size as a Keeper snapshot in a clean conversion.
• Newer operations through feature flags. ClickHouse negotiates a set of extra operations with Keeper that Apache ZooKeeper does not have (the flags are visible in a znode Keeper maintains), which lets a modern server do some work in fewer round trips against Keeper than against ZooKeeper.
• Same tools. system.zookeeper and the four-letter commands work on both, so monitoring carries over.

There is one reason to hold back, and it is organisational rather than technical. Keeper is built to serve ClickHouse. If the ensemble you want to retire also serves other applications (a message broker, a stream processor, exporters), you cannot move those with the converter, and the conversion will happily copy their trees anyway. The section on cleanup covers what to do about that.

What is in the ensemble, and how big it really is

Before you plan the cut-over, measure what you are moving. The ZooKeeper server exposes counters through the four-letter mntr command: znode count, ephemeral count, approximate data size and the state of the followers. On a production-sized cluster I saw several hundred thousand znodes and a logical data size of tens of megabytes, while the data directory (snapshots plus transaction logs) was tens of gigabytes. That gap is normal and it is the first lesson: the logical size predicts conversion time and Keeper memory; the directory size predicts how long copying takes. Conversion of a few hundred thousand nodes took seconds.

# needs a ZooKeeper or Keeper server
echo mntr | nc <host> <port> | grep -E 'zk_znode_count|zk_ephemerals_count|zk_approximate_data_size|zk_server_state|zk_synced_followers'

Write the numbers down; they never match exactly after conversion (see the verification section).

The converter

The tool is part of the clickhouse binary. Its entire interface, from clickhouse keeper-converter --help on 26.4:

Usage: clickhouse --zookeeper-logs-dir /var/lib/zookeeper/data/version-2
                  --zookeeper-snapshots-dir /var/lib/zookeeper/data/version-2
                  --output-dir /var/lib/clickhouse/coordination/snapshots
Allowed options:
  -h [ --help ]     produce help message
  --zookeeper-logs-dir arg       Path to directory with ZooKeeper logs
  --zookeeper-snapshots-dir arg  Path to directory with ZooKeeper snapshots
  --output-dir arg               Directory to place output clickhouse-keeper snapshot

Three directories, no dry run, no verification mode. It reads ZooKeeper’s on-disk state and writes one Keeper snapshot that every Keeper node is then started from.

What it reads. A ZooKeeper data directory holds snapshot files and transaction logs. ZooKeeper’s snapshots are fuzzy: they are serialised while transactions keep being applied, which is safe for ZooKeeper because recovery replays the log from the snapshot’s starting point with idempotent semantics. The converter loads the newest snapshot and then applies log records after it. That replay is the step to understand. Reading the converter’s source (ZooKeeperDataReader.cpp in the repository), two details stand out:

• If a multi-operation transaction in the log contains an operation that fails during replay, the whole transaction is skipped. That can happen legitimately when the fuzzy snapshot already contains the effect of one of its operations (a create that collides with a node the snapshot captured, a delete of a node it had already dropped). ZooKeeper’s own recovery tolerates that. The converter drops the other operations in the same transaction.
• The sequence counter of a parent node, the number that decides the name of the next sequential child, is taken from the parent’s cversion as stored in the snapshot.

What it writes. A Keeper snapshot. Nothing in the tool checks the result. If the input is inconsistent and there is no log to repair it from, it prints a warning about an inconsistent (fuzzy) snapshot, tries to apply logs, finds none, and exits with status 0.

In production. A rehearsal against a live production coordinator, converting a snapshot copied from a follower, produced a Keeper snapshot roughly a third the size of the one from an earlier rehearsal of a nearly identical tree. The conversion job reported success. Keeper then crashed on boot while deserialising the snapshot, reading past the end of the data. The compressed stream tested intact; the payload was logically short. The converter had printed its warning about the fuzzy snapshot, had no logs to replay because only the snapshot had been shipped, and wrote what it had. Re-running with the logs shipped alongside the snapshot, so that the replay could repair the fuzzy window, produced a good snapshot. The lesson was not “ship the logs”, it was “exit status 0 is not a success signal for this tool”.

The time bomb: sequence counters

A converted tree can be wrong in a way that does not crash anything. In a replicated table’s log/ node, every insert, merge or mutation is a sequential child (log-0000000123). ClickHouse creates the next entry by asking the coordinator for a sequential child, so the name comes from the parent’s counter. Cleanup removes consumed entries, but keeps a floor of them: on 26.4 the MergeTree setting min_replicated_logs_to_keep defaults to 10 (and max_replicated_logs_to_keep to 1000):

SELECT name, value FROM system.merge_tree_settings
WHERE name IN ('min_replicated_logs_to_keep', 'max_replicated_logs_to_keep', 'cleanup_delay_period');

┌─name────────────────────────┬─value─┐
│ max_replicated_logs_to_keep │ 1000  │
│ min_replicated_logs_to_keep │ 10    │
│ cleanup_delay_period        │ 30    │
└─────────────────────────────┴───────┘

Now combine the two facts. If the converted snapshot hands a log/ node a counter that is behind the newest retained entry, the next sequential create collides with an existing child (Node exists), a failed create does not advance the counter, and the cleanup that would free the slot is forbidden from deleting the last ten entries. The condition cannot resolve itself. Every insert into that table fails at the point where it writes its log entry, and no amount of waiting changes that.

In production. After a cut-over, the busiest replicated table on a staging cluster could no longer schedule merges, and then, days later, could not accept inserts: the ingestion pipeline’s jobs failed with Node exists on the table’s log path. An early diagnosis of “it will converge in a few hours” was wrong, because it missed the retention floor. The counter was stuck exactly at the number of the last retained entry. A full-tree comparison afterwards showed the mechanism: the Keeper side held a few dozen old log entries that ZooKeeper had already deleted, and lacked one deduplication-hash node that ZooKeeper had. The converted tree was a snapshot of the fuzzy window, not of a point in time. The repair, done with the table detached on every replica so nothing could write to the log while it ran, was to delete the consumed entries (after checking every replica’s log_pointer was past them), walk the counter forward by creating and deleting a dummy sequential child until the next suffix was past every pointer, and re-attach. The table behaved as if nothing had happened, and the runbook gained a gate.

You can detect the condition from a listing, before any ClickHouse server connects to the new tree. Here is the check on synthetic data. The listings are paths and cversion values, the kind you get from a tree dump of each side after a mapping step. The Keeper side has its log counter stuck on the newest child:

-- ran on clickhouse local; zk.tsv and keeper.tsv are (path String, cversion UInt64) listings
WITH children AS (
    SELECT extract(path, '^(.*)/log-\\d+$') AS parent, toUInt64(extract(path, 'log-(\\d+)$')) AS suffix
    FROM file('keeper.tsv', TSV, 'path String, cversion UInt64')
    WHERE path LIKE '%/log/log-%'
)
SELECT n.path AS log_node, n.cversion AS next_suffix, max(c.suffix) AS highest_child,
       if(n.cversion <= max(c.suffix), 'JAMMED: next create collides', 'ok') AS verdict
FROM file('keeper.tsv', TSV, 'path String, cversion UInt64') AS n
INNER JOIN children AS c ON c.parent = n.path
GROUP BY n.path, n.cversion;

┌─log_node─────────────────────────┬─next_suffix─┬─highest_child─┬─verdict──────────────────────┐
│ /clickhouse/tables/s1/events/log │        1002 │          1002 │ JAMMED: next create collides │
└──────────────────────────────────┴─────────────┴───────────────┴──────────────────────────────┘

Pointed at the ZooKeeper listing the same query reports ok (next suffix 1003, highest child 1002). Run it for every log node in the tree before you repoint a single server. A freshly converted tree from a clean snapshot passes it; a tree from a fuzzy one may not.

The cut-over plan

All of the above makes one rule: convert a snapshot that was written after writes stopped, and prove it was. The procedure that follows from it:

1. Freeze writers. Stop the ClickHouse servers (or at least every producer and the replication traffic), so the tree is static. Verify it from the ZooKeeper side: zk_znode_count identical on every node and unchanged across two reads some seconds apart, and zk_ephemerals_count at zero once the servers’ sessions have expired. In the rehearsals the ephemeral count drained to zero and the znode count dropped by exactly that number.
2. Write a marker. Create a znode outside the ClickHouse tree and record its czxid. It is the last transaction the ensemble will ever see before the cut-over, so a snapshot that contains it contains everything.
3. Force a fresh snapshot, and gate on it. ZooKeeper writes a snapshot when it starts, after replaying its log, and every snapCount transactions. Restart one follower: its startup snapshot now contains the whole frozen state. The gate is a comparison of numbers you can read from file names: the snapshot’s name carries the last transaction ID it covers, and it must be at least the marker’s czxid, and its modification time must be after the freeze. If it is not, stop.
4. If the restart did not produce a snapshot, force one. This is the surprise. A frozen ensemble has seen exactly one write (the marker). A production setting of snapCount in the low thousands means the restart replays a log that is far short of the threshold and ZooKeeper correctly declines to snapshot. In one attempt the gate failed for exactly this reason: the newest snapshot was a few transactions older than the marker. Both earlier “confirmations” that restart produces a fresh snapshot had been made against a live ensemble that was snapshotting every few seconds anyway. The fix was a small tool that issues enough no-op create/delete pairs on a throwaway path, outside the ClickHouse tree and with the path asserted before it runs, to push the transaction count past snapCount, then removes the throwaway nodes. The gate stays unchanged.
5. Make the source inert, then copy it. Only after the gate has passed, stop the ZooKeeper process on the copy source while keeping its data. Replacing its entrypoint with a long sleep keeps the host and its tooling available, so you can copy the directory with the same tools as before, while guaranteeing that nothing changes. Order matters: a stopped ZooKeeper does not write a snapshot on shutdown, so stopping it instead of restarting it gives the converter byte-identical input to the fuzzy snapshot you started with.
6. Convert. Run the converter on the copied directory. Because the fresh snapshot already contains every transaction, the log replay has nothing to do, and the converter’s weak path (replay) is never exercised.
7. Seed every Keeper node with the same output snapshot, and start Keeper on an empty log.
8. Verify before any ClickHouse server connects: the tree comparison and the counter check below.
9. Repoint ClickHouse at the Keeper nodes, start one server, and run a write-path probe: create a small replicated table, insert, select, drop with SYNC. A smoke test on a Memory table touches no replication log and tells you nothing, and one rehearsal’s smoke test made exactly that mistake. Then start the rest.
10. Soak with the old ensemble intact but stopped. Rollback is a configuration change that points ClickHouse back at ZooKeeper, which still holds the frozen state. It stops being a free rollback the moment ClickHouse has written to Keeper, so decide how long you keep it.

Bundling this with a server upgrade saves a restart per node; run them as two gated phases with separate rollbacks. See Upgrading ClickHouse safely.

Rehearsing against a live ensemble

You can rehearse the whole conversion against production without taking anything down, if you accept two things: you read from a follower, and you ship only the snapshot (plus any logs whose transaction IDs are not newer than it). With nothing to replay past the snapshot’s last transaction, the converter’s weak path does not run even though the source keeps changing underneath. The output is a valid snapshot of a past moment, which is exactly what a rehearsal needs to measure conversion time, snapshot size and Keeper start-up against the real tree.

The alternative with no converter

If you cannot freeze writers, stand up an empty Keeper, point the servers at it, and rebuild each table’s metadata from the parts on disk with SYSTEM RESTORE REPLICA. A table is read-only until it is restored:

-- needs a server; the table is read-only until the restore completes
SYSTEM RESTART REPLICA analytics.events;
SYSTEM RESTORE REPLICA analytics.events;

The converter preserves everything (log, dedup hashes, pending mutations) but needs the quiet window and the gates. Restoring loses the dedup history and the in-flight queue and must be done per table and replica, but needs no window. For hundreds of tables, the converter with its gates is the less error-prone route; for a handful, restoring is simpler.

The Keeper side of the configuration

Keeper is configured under keeper_server. The shape for a three-node ensemble is below; fill in your own hosts and keep the identifiers distinct on each node (needs a server).

<keeper_server>
    <tcp_port>9181</tcp_port>
    <server_id>1</server_id>
    <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
    <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
    <coordination_settings>
        <operation_timeout_ms>10000</operation_timeout_ms>
        <session_timeout_ms>30000</session_timeout_ms>
    </coordination_settings>
    <raft_configuration>
        <server><id>1</id><hostname>keeper-a</hostname><port>9234</port></server>
        <server><id>2</id><hostname>keeper-b</hostname><port>9234</port></server>
        <server><id>3</id><hostname>keeper-c</hostname><port>9234</port></server>
    </raft_configuration>
</keeper_server>

On the ClickHouse servers the zookeeper section changes from the old hosts to the three Keeper hosts on the client port, and nothing else. The converter’s --output-dir is the snapshot directory above; copy it to all three nodes with the ownership of the user Keeper runs as.

Verifying the copy, znode by znode

Counting is not verification. A Keeper node counts its nodes by different rules from ZooKeeper’s mntr (which itself disagrees by one with a recursive sum on the same server), so a flat difference of tens of nodes on a tree of hundreds of thousands says nothing. What does say something is a per-path comparison: dump the tree from both sides with clickhouse zookeeper-dump-tree (its options, from --help: -a addresses, -p start path, -c print ctime), or from a snapshot with clickhouse keeper-utils dump-state, and compare as data. clickhouse keeper-utils also offers snapshot-analyzer with --with-node-stats and --subtrees-limit, which ranks subtrees by size, useful for spotting which table dominates the tree.

Loading two listings into clickhouse local turns the comparison into one query. On synthetic listings, in which Keeper has a stale counter, extra old log entries and a missing deduplication node:

-- ran on clickhouse local
SELECT multiIf(z.path = '', 'only in keeper', k.path = '', 'only in ZooKeeper', 'cversion differs') AS verdict,
       if(z.path = '', k.path, z.path) AS path, z.cversion AS zk_cversion, k.cversion AS keeper_cversion
FROM file('zk.tsv', TSV, 'path String, cversion UInt64') AS z
FULL OUTER JOIN file('keeper.tsv', TSV, 'path String, cversion UInt64') AS k ON z.path = k.path
WHERE z.path = '' OR k.path = '' OR z.cversion != k.cversion
ORDER BY verdict, path
SETTINGS join_use_nulls = 0;

┌─verdict───────────┬─path────────────────────────────────────────────┬─zk_cversion─┬─keeper_cversion─┐
│ cversion differs  │ /clickhouse/tables/s1/events/log                │        1003 │            1002 │
│ only in ZooKeeper │ /clickhouse/tables/s1/events/blocks/b4          │           0 │               0 │
│ only in keeper    │ /clickhouse/tables/s1/events/log/log-0000000985 │           0 │               0 │
│ only in keeper    │ /clickhouse/tables/s1/events/log/log-0000000986 │           0 │               0 │
│ ...               │ ...                                             │         ... │             ... │
└───────────────────┴─────────────────────────────────────────────────┴─────────────┴─────────────────┘

(The real output has eight “only in keeper” rows, log-0000000985 to log-0000000992; the middle ones are elided here.) What to expect from a good conversion: no rows under replica, queue and block paths, and no rows at all under log nodes except where you can explain them. Interpreting the rows:

• Keeper-only log-* entries are old entries that ZooKeeper had deleted: the signature of a fuzzy snapshot with no replay.
• A ZooKeeper-only node under blocks/ is a deduplication hash lost in a skipped transaction.
• Leftover ephemeral nodes matter more than their size suggests: a replica’s is_active node is ephemeral, and a dead session’s copy can make a replica believe it is already running. Expect an ephemeral count of zero before servers connect.
• Counters (cversion): the check from the previous section.

Repeat the comparison a day after the cut-over, then check what ClickHouse itself thinks:

-- needs a server
SELECT database, `table`, is_readonly, is_session_expired, zookeeper_exception, queue_size, absolute_delay
FROM system.replicas
WHERE is_readonly OR is_session_expired OR zookeeper_exception != '' OR queue_size > 20;

Cleaning up what is left behind

A conversion copies the whole tree, not just ClickHouse’s part of it, and the old ensemble keeps its own copy of ClickHouse’s. Two clean-ups follow.

Foreign trees on the new ensemble. If the ZooKeeper ensemble was shared, the converted Keeper starts life with other applications’ roots: broker topology, stream-processing state, exporter bookkeeping. They are inert on Keeper, since nothing connects to them, and they cost memory and snapshot size; in one case a few dozen such roots held a few thousand znodes. Establish that they are dead by evidence rather than by name: no live broker registrations, no controller node (ephemeral, so present only while a broker is alive), modification times years in the past.

Do not delete them first. Rename them out of the way, soak, then delete. ZooKeeper has no rename, so a “rename” is a copy of the subtree under a new root followed by deletion of the original, done with a small client script. Move each foreign root under a prefix like /retired_<name>, leave the ClickHouse root and /zookeeper alone, and check system.replicas afterwards. After a soak of a few weeks, hard-delete in a list-driven pass with clickhouse zookeeper-remove-by-list (it reads node paths from standard input; -a takes the addresses).

The ClickHouse tree on the old shared ensemble. The old ensemble still holds a complete, now-orphaned copy of ClickHouse’s tree, hundreds of thousands of znodes sitting next to the applications that still use it. The same rename, soak and delete applies:

In production. The first attempt to walk several hundred thousand znodes through a forwarded connection from a laptop died with a connection loss in the middle of the dry run. The walk was sequential, one get_children per node, expected to take half an hour or more; the tunnel did not survive it, and nothing had been modified, which is the argument for always starting with a dry run. The fix was to run the script inside the network next to the ensemble. Size batches carefully: one multi-operation transaction is capped by the ensemble’s buffer size, one megabyte by default (jute.maxbuffer).

The on-disk footprint tells a second story. The same old ensemble’s logical state was tens of megabytes while its data directory held tens of gigabytes of snapshots and transaction logs. The directory size, not the node count, is what you reclaim by decommissioning, and if it keeps growing, check the purge settings (autopurge.snapRetainCount, autopurge.purgeInterval) before blaming the data. If an operator reconciles the old ensemble’s replica count, scaling it to zero will be reverted or rejected; remove it through the operator’s own objects, or pause the operator first.

Orphans inside the ClickHouse tree itself. Dropped replicated tables leave state behind if a replica was offline during the DROP, and decommissioned replicas leave their nodes. ClickHouse has statements for both, better than hand deletion:

-- needs a server
DROP TABLE analytics.events SYNC;                              -- removes the table's coordination state too
SYSTEM DROP REPLICA 'r2' FROM TABLE analytics.events;          -- a replica that no longer exists
SYSTEM DROP REPLICA 'r2' FROM ZKPATH '/clickhouse/tables/s1/events';   -- a path with no table left

To find candidates, compare what is registered with the tables you have (system.replicas.zookeeper_path):

-- needs a server; shard-level paths are children of each table's node
SELECT name, ctime, mtime, numChildren
FROM system.zookeeper
WHERE path = '/clickhouse/tables/s1'
ORDER BY mtime;

A path that no replicated table maps to, with an old mtime and few children, is a candidate. SYSTEM DROP REPLICA ... FROM ZKPATH refuses to drop the local replica’s own path, which is the guard you want.

Monitoring Keeper

The ensemble is now part of your database. These are the signals that I would alert on, ordered by how early they speak.

Four-letter commands. Send a four-character word to the client port: echo ruok | nc host 9181. Keeper restricts which words it answers through keeper_server.four_letter_word_allow_list (the name exists in the 26.4 binary; the defaults are in the Keeper guide). The useful ones:

Command	Answers	Alert on
ruok	imok if the process is alive	Anything else. A weak check: it answers even with no leader
mntr	key/value metrics	zk_server_state not leader or follower; zk_outstanding_requests rising; zk_avg_latency; zk_znode_count jumping; zk_synced_followers below quorum
srvr / stat	role, latency, connections	Role flapping
conf	the effective configuration	Drift between nodes
isro	ro if the node is read-only	ro
dirs	log and snapshot sizes	Disk growth

Readiness. ruok is the wrong readiness probe. Keeper also has an HTTP control endpoint with a /ready path that, in a one-node test, answered 200 with a hasLeader field, and answers 503 when a node has no leader; I prefer it to a shell probe. A readiness probe that runs ruok through a shell inside the container can fail by itself: when Keeper is the container’s init process nobody reaps the probe’s child, so every run leaves a zombie. At one run per ten seconds that is thousands of process IDs a day, and when the process limit was reached nothing could fork any more, including the probe. Use the HTTP endpoint.

From inside ClickHouse. The server is the best witness of Keeper’s health, because it is the one that gets hurt:

-- needs a server
SELECT name, host, port, index, is_expired, keeper_api_version FROM system.zookeeper_connection;

SELECT name, value FROM system.errors WHERE name ILIKE '%KEEPER%' OR name ILIKE '%COORDINATION%';

SELECT toDate(event_time) AS day, hostName() AS host, count() AS coordination_exceptions
FROM system.text_log
WHERE message LIKE '%Coordination::Exception%' AND event_date >= today() - 14
GROUP BY day, host ORDER BY day, host;

The last query is the most useful trend here. A healthy cluster logs a low, steady daily count of coordination exceptions per node; a storm shows up as a jump of two or three orders of magnitude. In one case the cause was not Keeper at all: routine maintenance on the hosts evicted a majority of a three-node ensemble at once, and every replicated table was read-only for hours. A three-node ensemble loses quorum with two simultaneous unavailabilities, so give whatever manages its nodes a disruption budget of one, anti-affinity so that no two members share a host, and a policy that keeps an autoscaler from evicting a member.

In system.metrics, the ones named for Keeper on 26.4 are KeeperAliveConnections and KeeperOutstandingRequests (meaningful where the server itself hosts Keeper), plus ZooKeeperSession, ZooKeeperSessionExpired and ZooKeeperRequest on the client side. clickhouse keeper-client is the interactive tool, with commands such as find_super_nodes (nodes with a very large number of children) and find_big_family (the biggest subtrees), the first place to look when a tree has grown unexpectedly.

Pitfalls

• Trusting exit status 0 and a successful boot. The converter reports success for a broken snapshot, and a broken snapshot can even boot. Gate on the snapshot ID and the marker, then compare paths and counters.
• A smoke test that skips the replication log. Test with a real replicated table: an insert, an OPTIMIZE and a mutation.
• A pending change that quietly reverts another. A repoint prepared weeks earlier on a branch that fell behind would have downgraded the servers’ image as a side effect of applying it. Rebuild it on the deployed base on the day, and prove with a render diff that the only difference is the coordination hostnames.
• Waiting for a stuck log counter to heal. It does not; look at the retention floor.
• Deleting foreign data first. Rename, soak, delete.
• Starting Keeper and servers in the same breath. Bring Keeper up, verify it, then start ClickHouse.
• Treating the old ensemble as free storage. After the cut-over its state is stale. Keep it intact but stopped for the rollback window only.
• Version skew. A Keeper a patch release away from the servers is fine; they speak a stable protocol, and Keeper ran a newer patch than its servers for weeks without trouble. Upgrade Keeper in its own change, one member at a time, checking quorum between members.