Build a CDC pipeline, step by step
The order I build a change data capture pipeline in, each step linked to the file in my public repositories that shows it done.
The decision it helps with
Where to start, and in what order, when a pipeline has to keep an analytics warehouse in step with a transactional database.
How to use it
- Work through the phases in order; each one rests on the one before it.
- Open the linked files to see a step done in a pipeline that runs, then adapt it to your own sources.
- Tick a step only when something checks it: a test, a query or an alert, not a belief.
- To see every phase running before you build your own, clone the cdc-starter template and run make demo.
The steps
Free to use and adapt under the Creative Commons Attribution licence: credit Matiwos Desalegn and link to this page.