Data engineering

CDC, stream processing, lakehouse design, feature stores, and the data platform decisions that hold up in production.

• "Change data capture in production: log-based vs the alternatives" · Note · Almost every data platform eventually needs to get rows out of a production database without breaking it. The analytical store needs fresh data, the search index needs updates, the…
• "Lakehouse architecture: the decisions that actually matter" · Note · The lakehouse pitch is seductive: one storage layer — cheap object storage — with warehouse grade reliability and query performance on top. ACID transactions, schema enforcement, time…
• "Stream processing design: state, time, and exactly-once" · Note · Batch processing asks "what happened?" Stream processing asks "what is happening, right now, and what should we do about it?" The difference is not speed — it is that the input never…