I used to think change data capture was mostly a tooling decision. Pick a connector, point it at Postgres, choose a destination, and deal with the details once the events start moving.
The more I look at real-time data systems, the less comfortable I am with that order. CDC moves changes from a production database into another system, but it does not automatically answer what should happen when a record is updated twice, a consumer falls behind, or a delete arrives after a long delay.
That is why the practical approach on Zamh Tech stood out to me. Its guides to how Postgres CDC works and production failure cases start with the database contract and the failure cases around it. That feels closer to how I would actually plan a pipeline.
CDC Is More Than A Faster Export
With a scheduled export, I already accept a delay. The system reads the source at a certain time, prepares a batch, and sends it somewhere else. That can be perfectly reasonable for reports that do not need to be current every minute.
CDC changes the timing. Instead of repeatedly asking the database for the current state, the pipeline follows changes as they happen. Inserts, updates, and deletes become events that another system can consume.
That sounds simple until I ask what the destination is supposed to represent.
Is it:
- The Current State Of Each Row?
- Every Change That Has Ever Happened?
- A Reconstructed History Of The Business Entity?
- A Stream For An Application Or Workflow?
- A Fresh Analytical Copy For Queries?
Those are different goals. A pipeline that is good at preserving every change may not be the easiest one to query as a current-state table. I would define this before choosing the connector or destination.
I Would Start With The Database Contract
The database contract is the part I would document first. It should explain what the source guarantees and what the downstream system can safely expect.
I would want clear answers to questions like:
- Which Tables Are Included?
- Which Columns Identify A Record?
- Are Updates Sent As Full Rows Or Partial Changes?
- How Are Deletes Represented?
- Can Events Arrive More Than Once?
- What Ordering Is Guaranteed?
- How Can A Consumer Resume After An Outage?
These questions are not just documentation work. They affect how the receiving system stores and processes the data.
For example, if an event contains only the changed columns, the consumer needs a reliable way to apply that partial update. If deletes are represented by a tombstone, the destination needs to preserve and process that signal instead of treating it as an empty record.
WAL And Replication Slots Are Operational Concerns
PostgreSQL change capture often depends on the write-ahead log, or WAL. The WAL records changes that happen in the database, which makes it possible for a consumer to read them later.
Replication slots help a consumer keep its place. That is useful, but it also creates an operational responsibility: if a consumer stops advancing, the source may need to retain more WAL than expected.
I would monitor that relationship from the first test rather than waiting for a production incident.
The checks I would keep visible include:
- Replication Slot Activity.
- Retained WAL Volume.
- Consumer Lag.
- Connector Health.
- Source Database Disk Growth.
- Last Successfully Processed Position.
The important thing is that a pipeline can be “running” while still falling dangerously behind. A green process status does not tell me whether the destination is receiving changes quickly enough.
The First Big Choice: Current State Or Event History?
One of the decisions I would make early is whether the destination should store the latest version of a record or preserve each change as an event.
A Current-State Table
This is convenient for many analytical queries. A customer, order, or product appears once with its latest values.
The challenge is applying changes correctly. The destination needs a stable key and a dependable way to decide which update wins.
An Append-Only Change Log
This keeps the history of changes. It is useful when I need to understand how an entity evolved or rebuild another view later.
The trade-off is that every query asking for the current state may need to interpret the change history first.
A Combination Of Both
In some systems, I would keep the raw change stream and maintain a derived current-state view for everyday queries.
That gives me more recovery options, but it also means more storage, more processing, and more decisions about reconciliation. I would not choose this automatically. I would choose it when the value of history justifies the added work.
Ordering Is Not A Detail
If two updates affect the same record, the destination needs to know which one happened first. That sounds obvious, but distributed systems can make ordering less straightforward than the source database suggests.
I would ask whether ordering is guaranteed per table, per key, per partition, or across the entire stream. Those are not interchangeable guarantees.
I would also identify the position or timestamp used to compare changes. A wall-clock timestamp may not be enough if clocks differ or if two events share the same time. A source log position may be more useful for replay, but it still needs to be carried through the pipeline.
The more I think about this, the more I see ordering as part of the data model rather than just a connector setting.
Duplicates And Retries Need A Clear Plan
At-least-once delivery is common because it is safer to retry than to silently lose a change. The consequence is that the destination may see the same event more than once.
I would design for that rather than hoping duplicates never appear.
Useful fields could include:
- A Stable Event Identifier.
- The Source Table And Primary Key.
- The Source Log Position.
- The Operation Type.
- The Event Creation Time.
- The Ingestion Time.
- The Connector Or Pipeline Version.
The exact fields depend on the system, but the destination needs enough context to detect duplicates and investigate a mismatch.
Idempotent processing can make retries safe. In simple terms, processing the same event twice should produce the same final result as processing it once. That is easier to achieve when the consumer has a stable key and an explicit version or position for the change.
Deletes Are Where Many Pipelines Get Uncomfortable
An insert is easy to imagine. An update is manageable. Deletes often expose an assumption that the destination is only a copy of the current database.
If a row disappears from Postgres, how does the analytical system know that it should remove or mark the corresponding record? If the source only exports rows that still exist, the destination may keep stale data forever.
I would decide whether the destination should:
- Remove The Record. Useful when the analytical copy should reflect only active source rows.
- Keep A Deletion Marker. Useful when the history of the entity matters.
- Move It To A Separate History Structure. Useful when operational queries and historical queries have different needs.
There is no universal answer. The important part is making the behavior explicit before the first delete arrives.
CDC Or ETL? The Question Is About Freshness And Complexity
CDC is not automatically better than scheduled ETL. I would choose based on the decision the data needs to support.
I would lean toward CDC when:
- A Downstream Workflow Needs Fresh Changes.
- The Source Changes Frequently.
- Reprocessing Full Tables Is Expensive.
- The Destination Needs To React To Inserts And Updates.
- Operational And Product Data Need A Near-Current View.
I would stay with scheduled ETL when:
- The Data Changes Slowly.
- A Daily Or Hourly Refresh Is Enough.
- The Transformation Is Easier To Express In Batches.
- The Team Wants A Simpler Operational Model.
- The Source Does Not Offer A Safe Change Stream.
The right answer can also be mixed. Some data can move through CDC while less time-sensitive tables continue through scheduled loads.
Where A Flexible Analytical Destination Fits
The destination matters because CDC does not only move neat, final tables. Over time, schemas change, new columns appear, and different consumers need different views of the same source activity.
This is where a flexible analytical system can be useful. RawTree, for example, is positioned around changing analytical data and SQL access, so I would consider it for a stream where the data model is still developing.
I would still define the CDC contract before sending anything there. Flexible storage does not remove the need to understand keys, deletes, ordering, retention, or permissions. It simply gives the team more room while the downstream model is taking shape.
The Checks I Would Run Before Production
I would test the pipeline with more than a successful insert. My test plan would include:
- A New Row Being Inserted.
- Several Updates To The Same Row.
- An Update With A New Column.
- A Row Being Deleted.
- Duplicate Delivery.
- A Consumer Restart.
- A Connector Outage.
- A Replication Slot Falling Behind.
- A Large Backfill.
- A Reconciliation Against The Source.
For each case, I would check both the final data and the operational evidence. Can I explain what happened? Can I resume from a known position? Can I tell whether the destination is complete, delayed, or missing a change?
That last question is important. A pipeline is not trustworthy just because it has moved data in the past. I need a way to prove that the source and destination still agree.
My Takeaway
The main lesson I took from Zamh Tech is that CDC should be designed around failure and meaning, not only movement.
Choosing a connector is the visible part. The harder work is deciding what a change means downstream, how the system recovers, which history is retained, and how the team notices that something has gone wrong.
I would start with the database contract, define the row identity and ordering rules, decide how deletes behave, and test lag before committing to a production architecture.
Once those decisions are clear, the tool choice becomes easier. A stream, warehouse, analytical database, or API can then be evaluated against the job it actually needs to do.
That is also why I would keep RawTree as one possible destination for evolving analytical data, rather than presenting it as the answer to every CDC project. The fit depends on the data, the freshness requirement, the queries, and the operational controls.
For me, the useful mindset is simple: move changes quickly, but understand exactly what the receiving system is expected to do with them.