Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

no it's not.

ETL is really database focused and batch focused , Extract, Transform, Load.

Data pipeline, is a combination of streams and batch. For example, you can implement a Data Capture using something like https://debezium.io/

Here's how Netflix solves it https://netflixtechblog.com/dblog-a-generic-change-data-capt...

Overview

"Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional techniques like dual-writes and distributed transactions [3][4]."



What your saying applys more to ELT, which is an efficient method when using batches. There's no reason why you can't stream ETL.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: