Teaching clickhouse-local to read ps aux
Replace grep | awk | sort with SQL: a request for an input format that reads whitespace-aligned CLI output, column names from the header row, last column greedy.
Source: github.com
I asked for an input format that reads the whitespace-aligned tables most shell tools print (ps aux, df, docker ps, ls -l): a run of spaces is one delimiter, the header row supplies the column names, and the last column is greedy so free-text tails like COMMAND survive. The goal is to make ps aux | clickhouse local -q "..." replace grep | awk | sort. The issue includes a roughly 150-line prototype whose row counts matched awk on ten real commands.
It is open, labelled feature, comp-formats and external, with no comments and no pull request. The general version, #34006 (schema inference for freeform text), is closed.
On 26.4 the nearest format, CustomSeparatedIgnoreSpacesWithNames with a space delimiter, throws BAD_ARGUMENTS, and Regexp cannot read the header. What works is splitting in SQL, which keeps the greedy tail with arraySlice:
SELECT f[1] AS user, toUInt32(f[2]) AS pid, toFloat64(f[3]) AS cpu,
arrayStringConcat(arraySlice(f, 4), ' ') AS command
FROM (SELECT splitByWhitespace(line) AS f
FROM file('/tmp/ps.txt', 'LineAsString', 'line String'))
WHERE f[1] != 'USER';
root 1 0.5 /sbin/init splash
me 22 3 sleep 10
That needs you to know the shape of ps in advance, which is what the request wanted to avoid. See also query inline data with the format table function.
Discussion
Comments live in GitHub Discussions. Sign in with GitHub to join in.