Real-time, Exactly-once Data Ingestion from Kafka to ClickHouse at eBay

Real-time, Exactly-once Data Ingestion from Kafka to ClickHouse at eBay

eBay depends on Kafka to solve the impedance mismatch between rapidly arriving messages in event streams and efficient block insert into ClickHouse clusters. Naïve loading procedures from Kafka to ClickHouse generate non-deterministic blocks, leading to data loss and incorrect results…