Blog· Tested on ClickHouse 26.4

Which rows did input_format_allow_errors_num throw away?

Allowing parse errors on import is great until you need to know which rows were dropped. I asked in 2020; an error file landed in 22.9, and the issue stayed open until 2026.

Source: github.com

input_format_allow_errors_num lets a CSV or TSV load tolerate up to N bad rows, but ClickHouse then skipped them silently: you did not know how many were dropped, let alone which. Finding them meant rerunning with a limit of zero and fixing one row per trip. I asked in 2020 for the rejected rows to be reported to stdout or stderr.

A maintainer replied the same day with a better design: a fallback table with one String column, plus a rejected-row counter in ProfileEvents. The idea then sat for years. What shipped is neither: PR #40516, “Record errors while reading text formats (CSV, TSV)”, merged 2022-09-14 and released in 22.9, adds input_format_record_errors_file_path, which writes every rejected row to a file with its row number and the reason; errors_output_format picks the file format (CSV by default). The issue itself stayed open until 2026-06-12, when someone noticed it had been implemented all along.

On 26.4, with 2,oops and four,80 as the bad rows:

SELECT * FROM file('events.csv', 'CSV', 'user_id UInt64, duration_ms UInt32')
SETTINGS input_format_allow_errors_num = 10,
         input_format_record_errors_file_path = 'rejected.csv';

The load returns the good rows, and rejected.csv holds a timestamp, database and table when known, the row offset, the parser’s reason and the raw text. The ProfileEvents counter and any stdout option I did not find.

feature-requestingestionformats

Discussion

Comments live in GitHub Discussions. Sign in with GitHub to join in.