Slowly draining the backlog
The Hidden Cost of CDC Pipelines: How Small Files Create an S3 Request Bomb
In Why We Redesigned Our CDC Pipeline with Debezium and Flink, we introduced our Debezium + Flink-based CDC pipeline. Debezium captures database changes, routes them through Kafka, and Flink writes Parquet files to S3 every 5 minutes. The pipeline worked well for over a year. Data consistency was verified, and operations were stable. The problem remained invisible until we dug into our S3 costs as part of a data platform TCO analysis. ...