When did this table last receive data? A last_insert_time that was never built
Spotting a stuck ingestion job across hundreds of tables needs the last INSERT time. I asked for a column in system.tables or system.parts; the answer was a workaround that is still the answer.
Source: github.com
To spot stuck ingestion across hundreds of tables I needed the time each table last received data. max(event_time) per table is the wrong tool: it scans data, some tables have no time column, and it measures time in the data, not arrival time. I asked for last_insert_time in system.tables or system.parts.
The maintainer explained why it is hard: parts do not record insert times, and a trigger-style approach breaks with replication. A workaround from a user, system.part_log filtered on NewPart and DownloadPart, was rejected as incorrect, since a download (replica repair, fetched merged parts, network catch-up) is not an INSERT. I narrowed it to NewPart events, queried through a Distributed table over system.part_log so every replica is covered:
SELECT database, table, max(event_time)
FROM part_log_distributed
WHERE event_type = 'NewPart' AND event_date > now() - INTERVAL 5 DAY
GROUP BY database, table;
In April 2023 the maintainer replied that the trick “could suit as a workaround” and closed the issue. No column was added, so I treat the request as unmet.
The workaround has costs: part_log must be enabled, retention is limited, and a table silent for longer than the log keeps will not appear at all, which is exactly the case you are hunting. See inserts in the last 15 minutes and how often is a table inserted into.
Discussion
Comments live in GitHub Discussions. Sign in with GitHub to join in.