# Mati Data: full text > The public content of https://matidata.ethioneural.com/ as plain Markdown, for AI assistants: who Matiwos Desalegn is, roles, services, credentials and the full text of each public case study. The short index is https://matidata.ethioneural.com/llms.txt. I help teams design, build and govern data platforms they can trust. Senior Data Engineer at Kifiya Financial Technology in Addis Ababa, running lakehouse, CDC and warehouse platforms for digital lending since 2024. I take data work from architecture and governance to production pipelines and analytics. AI engineering is my second discipline. 5+ years building with data and software (since 2021), including production data engineering at scale since 2024. Availability: Open to conversations about opportunities where I can make a meaningful contribution. Email: matidesalegnece@gmail.com (personal), mathiwossilvio@gmail.com (work) Book a thirty-minute call: https://cal.com/matiwos-desalegn-b2fps4/matidata, or through Calendly: https://calendly.com/matidata/30min Contact page: https://matidata.ethioneural.com/contact ## Roles ### Senior Data Engineer, Kifiya Financial Technology PLC Jul 2026 to present, Addis Ababa, Ethiopia. Since July 2026 I have worked on the newest layers of Kifiya's data platform: near real-time replication and Gold reporting models for a smart mobility partner, the lakehouse governance plane and the deployment path for dbt. I still run the orchestration and feature pipelines I built as a Data Engineer. ### Data Engineer, Kifiya Financial Technology PLC Oct 2024 to Jun 2026, Addis Ababa, Ethiopia. I joined while the data platform was still being built. I stood up its first lakehouse, the Redshift warehouse and the ML feature store for multiple institutional partners, moved orchestration to Airflow on EKS, and built GenAI services on top. ### Software Developer, HST Consulting PLC Nov 2023 to Oct 2024, Addis Ababa, Ethiopia. Frontend developer on a SaaS HR and payroll product, working with the backend team on the APIs behind it. ## Education - Kifiya AI Mastery Training Program (KAIM): data engineering and machine learning intensive, 10 Academy (Jun 2024 to Aug 2024) - BSc, Electrical and Computer Engineering, Wolaita Sodo University (Sep 2018 to Jul 2023) ## Services ### Data architecture Target-state designs for lakehouses and warehouses that fit the sources, the team and the budget you actually have, written down as decisions you can defend later. - Platform and enterprise data architecture: medallion lakehouse, warehouse and serving layers - Source integration patterns: log-based CDC, high-watermark extracts and dump restores - Technology choices with the tradeoffs on record, such as when a lighter CDC tool serves better than a stream processor - Data models: dimensional marts, slowly changing dimensions and one-table reporting views ### Data-intensive systems design Designs that hold up under retries, replays and partial failure, reasoned from how storage engines, logs, replication and transactions actually behave. - Design reviews in the vocabulary of Designing Data-Intensive Applications: replication, sharding, transactions, consistency, batch and streams - Change data capture from database logs, ordered by the source's log position rather than arrival time - Exactly-once results from at-least-once delivery: idempotent writes, natural keys, and watermarks committed with the data - Storage engine and table-format choices for the access pattern: B-tree row stores, merge-tree engines such as ClickHouse, and Iceberg tables - Failure analysis that finds the quiet failures, the ones that produce wrong data without an error ### Data engineering Production pipelines that fail loudly instead of quietly: orchestrated, tested, idempotent and cheap to rerun. - Airflow on Kubernetes with retries, data-quality tasks and alerting - Change data capture with exactly-once loads into Iceberg and the warehouse - dbt on Trino and Redshift, tested and promoted through CI/CD environments - Incident work when a production database is under strain: scheduling, restores and recovery ### Data governance and quality Governance built into the platform rather than added in meetings: contracts at model boundaries, lineage on every run, and access rules enforced at query time. - Data contracts and quality gates that stop a bad model before it reaches a dashboard - Lineage with OpenLineage and Marquez, cataloging and PII tagging with OpenMetadata - Role-based access and column masking with Apache Ranger and Trino - Quality controls and pipeline service levels for the datasets the business depends on ### Analytics engineering and BI Gold layers that answer the business question in one table, so analysts and dashboards stop joining raw data at query time. - Reporting masterviews and semantic marts in dbt - Serving from Superset, Redshift and ClickHouse with secured self-service access - Materialized-view scheduling that keeps operational reporting fast ### Enablement and training Leaving a team able to run what we built: clear handovers between data science and data engineering, documentation that answers the 2 a.m. question, and teaching that sticks. - Handover templates that turn a data-science feature request into an engineering spec - Internal lectures on machine learning for analysts and engineers - Runbooks, lineage and tests left in place so the platform does not depend on one person - Community workshops on machine learning, web and mobile development ### AI and ML on your data Putting models on top of a platform that can feed them, from feature stores to retrieval over your own documents. - Feature stores shared by training and serving, behind low-latency decision APIs - Retrieval-augmented assistants over internal documents, with local models where needed - Ethiopian-language NLP: embeddings, OCR and small-model evaluation ## Ways to work together Every engagement is scoped and priced after a short call; there is no public price list. ### Architecture review A focused review of an existing platform or a planned design, ending in written recommendations you can act on. - Best fit: You have a platform or a plan and want a second opinion before committing to it. - What it covers: Sources, pipelines and models, and who owns each; Platform choices, running costs and where the money goes; Data quality, lineage and access controls; The design or migration plan on the table - What you get: A current-state map: sources, flows, owners, and where the data goes wrong today; A target-state diagram that traces each business question back to its sources; Decision records for every significant choice, with the trade-off on record; A phased roadmap: one slice end to end first, then source by source; The risks and running costs, ranked - Typical timeline: Usually two to three weeks, depending on the size of the estate and access to people. - Not included: Building the changes. The review ends in written recommendations; implementation is a separate engagement. ### One-day design review One focused day on a design, a migration plan or a platform you are about to commit to, ending in written notes. - Best fit: One decision is close, and you want an experienced second look before you commit to it. - What it covers: The design, plan or platform you share before the day; The decisions behind it, and the risks they carry; Alternatives worth a second look - What you get: A working session with your team on the day; Written notes: what holds up, what to change, and what to check next; A short list of decisions to write down before you build - Typical timeline: One day, with the written notes after it. - Not included: A full current-state map or a roadmap; that is the architecture review. ### Platform build Designing and building a lakehouse, warehouse or CDC pipeline together with your team. - Best fit: You need a production data platform and the people to run it afterwards. - What it covers: The architecture, with a decision record for each significant choice; Ingestion and change data capture from your sources; Models, tests and data contracts; Monitoring, alerts and runbooks; Handover and pairing with your team - What you get: One source running end to end into a trusted table and dashboard, then the next; Tests, alerts and runbooks for every stage that ships; Decision records, so the reasons outlive the project; A handover your team signs off, with the code and documents in your repositories - Typical timeline: Scoped slice by slice; the first slice, one source end to end, usually takes a few weeks. - Not included: Running the platform for you after the handover, or on-call cover. ### Governance and quality setup Data contracts, lineage, cataloging and access control added to a platform you already run. - Best fit: Your data exists, but nobody fully trusts it. - What it covers: Data contracts at the boundaries between models and teams; Tests and freshness checks on the tables people rely on; Lineage on every run, and a catalog people can search; Access rules enforced where the data is queried - What you get: Contracts and tests that fail the build before bad data ships; Lineage and a catalog for the tables in scope; Access rules, with personal data tagged; A short playbook your team can extend - Typical timeline: Usually three to six weeks for a first domain, then domain by domain. - Not included: Rebuilding the platform. This works with the stack you already run. ### AI readiness check A check of whether your data can carry the AI use cases you have in mind, before you fund them. - Best fit: AI use cases are lined up, and nobody can yet say whether the data behind them is good enough. - What it covers: Data quality and ownership behind each use case; Definitions and lineage; Access, privacy and personal data; Platform capability and running cost; Operating risk once a model is in use - What you get: A readiness scorecard for each use case; A risk register, with an owner for each risk; The constraints each use case has to respect; A sequenced roadmap: the data fixes first, then the capability - Typical timeline: Usually two to three weeks, depending on the number of use cases and access to people. - Not included: Building or tuning models, choosing vendors, deploying to production, or legal advice. ### Team workshop A hands-on workshop for your engineers on a running example, so the practice sticks after the day ends. - Best fit: Your team needs to learn a practice by building it, not by watching slides. - What it covers: A topic from the outlines below, or one shaped to your stack; Exercises on a running example: your platform, or one of my public pipelines; Material your team keeps and can reuse - What you get: An outline agreed before the day; Hands-on exercises, with a working setup for each participant; The slides, the exercises and a reading list; A short note afterwards on what to practise next - Typical timeline: Half a day to two days, depending on the topic and the group. - Not included: Certification, or support after the workshop unless it is agreed separately. ### Senior or lead role A full-time role leading data architecture and engineering, remote with a team in Europe or based in Africa. - Best fit: You are hiring a data architect or a senior data engineer. ## Workshops ### Change data capture, end to end Data engineers moving from nightly batch loads to streaming changes out of a transactional database. - You will be able to: Set up logical replication and a Debezium connector safely; Land changes in ClickHouse with ordering and deletes handled; Measure lag against a heartbeat and reconcile row counts; Recover from a stopped connector without losing a change - Outline: When change data capture is worth it, and when it is not; The database side: keys, replica identity, publications and slots; Debezium, the event format and the topic; Landing and current state in the warehouse; Monitoring, alerts and a recovery drill - Length: One day, hands-on, with a running stack on each laptop ### dbt models people can trust Analytics and data engineers who own dbt models that other teams depend on. - You will be able to: Choose the grain and the key of every model, and test both; Put each test at the layer where the problem first appears; Declare freshness on the sources, not only on the marts; Gate merges in CI on the tests and the project's conventions - Outline: Grain, keys and what a model promises; Tests by layer: staging, intermediate and marts; Source freshness, and why it beats checks on marts; Shared logic in one macro; CI gates and documentation - Length: Half a day to a day ### Observability for data pipelines Teams whose dashboards say everything is healthy while the data is late or missing. - You will be able to: Pick the signals that matter: lag, freshness, row drift and connector state; Turn what is in the tables into metrics with a small exporter; Write alert rules with runbooks, and test them; Run a failure drill and read the signals - Outline: What infrastructure metrics miss; Signals from the data itself; Alert rules, runbooks and their tests; Dashboards for the person on call; A failure drill - Length: Half a day ## Evidence, question by question ### Can you design and build a data pipeline end to end? Yes. Two public pipelines run from a source API through PostgreSQL, Debezium and ClickHouse to tested dbt marts, with orchestration and monitoring around them, and each design decision is written up. - [World Bank CDC analytics](https://matidata.ethioneural.com/work/wb-cdc-analytics) - [Kiva loan CDC analytics](https://matidata.ethioneural.com/work/kiva-microfinance-loan-cdc-analytics) - [The pattern, stage by stage](https://matidata.ethioneural.com/#lineage-heading) - [CDC Starter template](https://github.com/matidesalegn/cdc-starter) ### Do you test and monitor what you build? Every figure in the evidence ledger on the home page links to the file behind it: the tests, the alert rules and the monitor that reconciles row counts between the source and the warehouse. - [Evidence you can check](https://matidata.ethioneural.com/#proof-heading) - [World Bank alert rules](https://github.com/matidesalegn/wb-cdc-analytics/blob/main/observability/prometheus/alerts.yml) - [Kiva alert rules](https://github.com/matidesalegn/kiva-microfinance-loan-cdc-analytics/blob/main/config/grafana/provisioning/alerting/rules.yml) ### Can you explain the trade-offs behind a design? Each case study has a section on decisions and trade-offs, and the architecture page sets out how I choose between a warehouse, a lake and a lakehouse, and between ETL, ELT and change data capture. - [The choices I make first](https://matidata.ethioneural.com/architecture#choices-heading) - [The principles I design by](https://matidata.ethioneural.com/architecture#principles-heading) ### Do you document what you build? Each public pipeline has its requirements, a data catalogue of every table and column, the naming conventions the code follows, and a design report with the trade-offs. - [World Bank data catalogue](https://github.com/matidesalegn/wb-cdc-analytics/blob/main/docs/data_catalog.md) - [World Bank naming conventions](https://github.com/matidesalegn/wb-cdc-analytics/blob/main/docs/naming_conventions.md) - [Kiva requirements](https://github.com/matidesalegn/kiva-microfinance-loan-cdc-analytics/blob/main/docs/requirements.md) - [Kiva data catalogue](https://github.com/matidesalegn/kiva-microfinance-loan-cdc-analytics/blob/main/docs/data_catalog.md) ### Do you work on AI as well as data? Yes, as a second discipline that sits on good data: a retrieval assistant over local documents, OCR for Ethiopic script, and research on quantising small Amharic language models. - [Local RAG assistant](https://matidata.ethioneural.com/work/local-rag-assistant) - [Amharic OCR for Ethiopic script](https://matidata.ethioneural.com/work/amharic-ocr-ethiopic-script) - [Amharic small language models](https://matidata.ethioneural.com/work/amharic-slm-quantization) ### Are your credentials real? Each one links to the issuer's own page, so you can check it rather than take my word for it. The DataCamp tracks are career tracks, not vendor certifications. - [Associate Data Engineer in Databricks](https://www.datacamp.com/completed/statement-of-accomplishment/track/c30477348cf0aa594de48a9a629bd6f968fded0a) - [Associate Data Engineer in Snowflake](https://www.datacamp.com/completed/statement-of-accomplishment/track/ab2454e5c82202444ba2f115d9d6d30b4b82ca83) - [AWS badge on Credly](https://www.credly.com/badges/081eb812-b18d-40de-ab6c-baff50ad63e8) - [All credentials](https://matidata.ethioneural.com/about#credentials-heading) ### Have you led or taught other engineers? I led the Google Developer Student Club at Wolaita Sodo University, and my community and teaching work is listed on the About page. - [Community and teaching](https://matidata.ethioneural.com/about#community-heading) ### What do people you have worked with say? A recommendation from a senior data engineer I worked with is on the About page, and every recommendation I have received is on LinkedIn. - [Recommendation](https://matidata.ethioneural.com/about#recommendations-heading) - [Recommendations on LinkedIn](https://www.linkedin.com/in/matiwos-desalegn/details/recommendations/) ### Where is your production experience? On the CV, role by role, with the platforms I built on and what they were for. - [Full CV](https://matidata.ethioneural.com/cv) - [Data engineering CV](https://matidata.ethioneural.com/cv/data-engineering) ## Talks Mati can give ### Change data capture that tells the truth: lag, drift and deletes Most CDC demos stop when the first row arrives. This talk starts there: how to tell a quiet table from a stopped connector, why deletes and ordering break naive pipelines, and how reconciling counts between the source and the warehouse catches the gaps a healthy dashboard hides. Built on two public pipelines, with the code on screen. - For: Data engineers and platform teams running or planning change data capture. - Length: Thirty to forty-five minutes, or as a workshop ### One CDC pattern, two builds: what changes between Airflow and Dagster The same pipeline shape, from PostgreSQL through Debezium and ClickHouse to dbt, built twice with different orchestration and monitoring. What stayed the same, what each tool made easier, and which choices mattered more than the tool. - For: Meetups, and teams choosing an orchestrator or a monitoring approach. - Length: Thirty minutes ### Is your data ready for AI? The checks before the model The questions I would settle before funding an AI use case: who owns each source it needs, whether the definitions agree, what personal data the model will see, what it will cost to run, and how its quality will be watched once people rely on it. - For: Leaders and teams planning AI on top of the data they already have. - Length: Thirty minutes ### Choose the shape before the tools: warehouse, lake or lakehouse A few choices set most of a data platform's cost and risk before any code is written: the storage shape, the integration style, who owns the data, and where definitions live. How I make each one, and the signals that should change the answer. - For: Architects, engineering leads, and university audiences. - Length: Thirty to forty-five minutes ## How Mati uses AI I use AI assistants every day, mostly Claude Code, as a fast pair programmer and editor. I decide what gets built and why, and nothing an assistant produces ships until it has been checked. ### Where it helps - Drafting code and tests from a design I have already written down - Reviewing my changes for mistakes before they are committed - First drafts of documentation, which I then edit - Finding my way around an unfamiliar library or codebase - Checking my writing against my own style and confidentiality rules ### Where I do not use it - Choosing an architecture or a trade-off: those decisions, and the reasons for them, are mine - Deciding whether data is correct: tests, reconciliation and alerts do that, not a model's opinion - Employer or client code, data and documents: they stay out of my personal AI tools - Claims about my experience: every figure on this site comes from my own records, not from a model ### How the output is checked - Every figure on this site sits on an allowlist with its source, and the build fails on any figure that does not - Before anything deploys, the site runs content checks, a confidentiality audit and hundreds of browser tests - Code changes come with tests, and the public pipelines show their checks in their repositories An AI assistant helped build this site and draft some of its text. I review every page before it goes live, and every figure on it links to its source. ## Free checklists and guides - [CDC pipeline checklist](https://matidata.ethioneural.com/resources/cdc-pipeline-checklist): What a change data capture pipeline needs before people rely on it: keys, ordering, deletes, lag, reconciliation and recovery. - [Build a CDC pipeline, step by step](https://matidata.ethioneural.com/resources/cdc-build-guide): The order I build a change data capture pipeline in, each step linked to the file in my public repositories that shows it done. - [Platform choice scorecard](https://matidata.ethioneural.com/resources/platform-choice-scorecard): Score the four choices that set a data platform's shape: storage, integration, ownership, and where definitions live. - [AI readiness checklist](https://matidata.ethioneural.com/resources/ai-readiness-checklist): Questions to answer before funding an AI use case: whether the data under it, its owners and its controls can carry it. ## Architecture choices ### Warehouse, lake or lakehouse? A lakehouse on an open table format when BI and machine learning read the same data. A warehouse when the questions are known, the data is structured and the team is small. - Warehouse: The questions are known, the data is structured, and analysts need fast SQL more than raw history. - Lake: Raw and semi-structured data has to be kept cheaply for uses nobody has named yet. - Lakehouse: BI and machine learning share tables, so open files such as Iceberg or Delta need warehouse guarantees. ### ETL, ELT or change data capture? ELT with dbt for analytics, so raw data lands once and every transformation is versioned and tested. Change data capture from the database log when freshness or deletes matter. - ETL: Sensitive fields must be dropped or masked before data leaves the source, or the target cannot do the work. - ELT: The warehouse can transform at scale, and analysts gain from raw history and tested dbt models. - Change data capture: Freshness matters, deletes must reach the warehouse, or polling would load the source database. ### Central, hub-and-spoke or domain ownership? Start central, with data contracts at the boundaries so the team does not become the bottleneck later. Move to hub-and-spoke as domains gain their own analysts. A data mesh is an organisational change, not a tool. - Central: One team runs one platform for a modest number of sources and consumers. - Hub-and-spoke: A platform team keeps the infrastructure and standards while domain teams build their own models. - Domains (data mesh): Domains can own data products end to end, with contracts, documentation and support. ### Where do definitions live? In one place, enforced where data crosses a boundary: a catalog for what exists, a semantic layer for what a number means, lineage for where it came from, and contracts that fail fast. - Catalog: People cannot find the data, or cannot tell who owns it. - Semantic layer: Teams report different numbers for the same metric. - Lineage: A change upstream breaks something downstream and nobody can say what. - Data contracts: A producer's change should fail its own build, before consumers see wrong data. ## Credentials - Associate Data Engineer in Databricks (career track), DataCamp (2026-09) - Associate Data Engineer in Snowflake (career track), DataCamp (2026-07) - Associate Data Engineer in SQL (career track), DataCamp (2026-07) - Inclusive Open Source Community Orientation (LFC102), The Linux Foundation (2026-08) - Intermediate SQL, DataCamp (2026-07) - Apache Airflow Essential Training, LinkedIn Learning (2026-02) - Data Engineering Pipeline Management with Apache Airflow, LinkedIn Learning (2026-02) - PySpark Essential Training: Introduction to Building Data Pipelines, LinkedIn Learning (2026-02) - Advanced NoSQL for Data Science, LinkedIn Learning (2026-02) - Problem-Solving Strategies for Data Engineers, LinkedIn Learning (2026-02) - Collaboration Principles and Process, LinkedIn Learning (2026-02) - Tech Career Skills: Effective Technical Communication, LinkedIn Learning (2026-02) - Designing Highly Scalable and Highly Available SQL Databases, LinkedIn Learning (2026-01) - Data Engineering Foundations, LinkedIn Learning (2026-01) - Introduction to Amazon Redshift: Data Management Essentials, LinkedIn Learning (2026-01) - Learning Apache Airflow, LinkedIn Learning (2026-01) - Advanced SQL Project: Design and Manage a Database, LinkedIn Learning (2026-01) - Agentic AI Design Patterns for GenAI and Predictive AI, LinkedIn Learning (2026-01) - Enterprise Architecture in Practice, LinkedIn Learning (2026-01) - Introduction to Airflow in Python, DataCamp (2025-11) - Fundamentals of Text Data Manipulation, CodeSignal (2025-09) - Claude Code in Action, Anthropic (2025-07) - AWS Educate Machine Learning Foundations, Amazon Web Services (2025-06) - Introducing Generative AI with AWS, Udacity (2025-06) - Data Analysis Fundamentals, Udacity (2024-07) ## Case studies ### World Bank CDC analytics: PostgreSQL to ClickHouse, orchestrated by Airflow URL: https://matidata.ethioneural.com/work/wb-cdc-analytics Stack: PostgreSQL, Debezium, Redpanda, ClickHouse, dbt, Apache Airflow, Prometheus, Grafana, Docker Compose, GitHub Actions, Python, Make Updated: 2026-10 Note: built in three days as a take-home technical assessment A three-day take-home assessment, built on the World Bank's public Indicators API: PostgreSQL, Debezium into Redpanda, ClickHouse with dbt marts, Airflow, and Prometheus and Grafana, started with one command. In plain words: When a row changes in the source database, this pipeline copies the change into the analytics warehouse within seconds, keeps the reports in step, and checks that nothing was lost on the way. #### Context I built this in three days in August 2026 as the take-home assessment for a senior data engineering role, and kept it public because most of my change-data-capture work lives in private employer repositories. It is a reference implementation of how I think a CDC analytics pipeline should be put together and, more to the point, how it should fail. It ingests a public REST API, the World Bank Indicators API, into PostgreSQL, streams every change out of the write-ahead log with Debezium into Redpanda, lands the events in ClickHouse, and builds a staging layer, an analytics mart and a machine-learning feature table with dbt. Airflow orchestrates, Prometheus and Grafana watch, GitHub Actions tests, and `make demo` starts all of it. The data comes from World Bank Open Data, and the project is not affiliated with or endorsed by the World Bank. I optimised for decisions I can defend and for failures that announce themselves. Most of what I have debugged in production was not a component that crashed but one that kept running while quietly doing the wrong thing. #### Constraints - One command, idempotent: a second `make demo` exercises the change-detection and incremental paths instead of repeating the first run. - Laptop-sized: every image pinned, memory capped explicitly, ports bound to loopback on non-default numbers so the stack cannot collide with a database already running on the machine. - Scope written down before code: what is built, what is deliberately left out and why. No Schema Registry, no clustering, no history tables, no Alertmanager routing, no BI layer. - Every stage verifiable by a script with a meaningful exit code, because prose drifts and a script does not. #### Architecture The World Bank CDC pipeline: data moves top to bottom, and orchestration and observability run across every stage. Stages, from source to serving: - World Bank API: Public, paginated, no key - Python client: Retry with jitter, response contract - PostgreSQL 17: Change-detecting upsert in one commit - Debezium: Reads the write-ahead log through pgoutput - Redpanda: Kafka-compatible event log - ClickHouse: Raw JSON, typed views, ReplacingMergeTree by LSN - dbt: Staging, analytics mart and feature table - Orchestration, across every stage: Airflow pipeline and liveness DAGs; GitHub Actions CI - Observability, across every stage: Prometheus and Grafana; 10 alert rules, each with a runbook Ingestion is a small Python client with pagination and retry with jitter, behind a contract that checks the shape of the response rather than its status code: this API returns errors inside successful responses and some transient failures as client errors. Rows load into PostgreSQL 17 through a change-detecting upsert, and the data, the audit row and the watermark commit in one transaction, so a recorded position can never claim work that rolled back. Debezium 3.6.1 reads the WAL through `pgoutput` over an explicit publication and writes flattened events to Redpanda, with the operation, the LSN and a deleted flag carried alongside the row. ClickHouse reads each topic through a Kafka engine table with one raw JSON column. Materialized views do the typing and write twice: to an immutable event log, and to `ReplacingMergeTree` landing tables versioned by the source LSN, so the newest row follows the source's commit order rather than arrival order. dbt builds staging views that apply `FINAL` and the tombstone filter from exactly one macro, then two dimensions, an incremental fact and a feature table. Airflow runs the pipeline with quality gates in the critical path plus a separate liveness DAG. Prometheus scrapes ClickHouse and Redpanda natively, one exporter covers the signals that live in tables, and Grafana is provisioned from the repository. #### Decisions and tradeoffs Three decisions I would defend, each preventing a failure that produces no error at all: - The natural key is the primary key. Under the default replica identity a delete event carries only the primary key, so a surrogate key would make deletes arrive downstream as an integer with no way to identify the business entity. Full replica identity would fix that by multiplying WAL volume; the key choice fixes it for free. - The upsert is change-detecting. Without `WHERE source_hash IS DISTINCT FROM EXCLUDED.source_hash`, every re-ingest rewrites every row and each no-op update emits a CDC event, so the change stream describes the scheduler instead of the data. - The Kafka engine reads raw JSON and typing happens in the views. Typed Kafka-engine columns look cleaner and are a trap: one parse failure fails the block, offsets are never committed and the consumer retries forever with no error reaching anyone. With raw JSON the consumer cannot fail, and the pipeline degrades to nulls instead of stalling. Two more are worth naming. CDC lag is measured against a Debezium heartbeat, not a business table, because an idle table reports the same number as a stopped connector. And Great Expectations is substituted, not skipped: dbt tests and pytest cover the relational checks, explicit assertions guard the boundary before data lands, and the design notes say where I would add Great Expectations first, at the feature boundary where the useful checks are distributional. Redpanda instead of Kafka with ZooKeeper, the Kafka engine instead of a Connect sink, and plain JSON instead of a Schema Registry each keep the stack small and inspectable, and each cost is written down. #### Outcome - 2,970 rows ingested across 36 paginated requests; a second run writes nothing. - Latency of about 3 seconds from PostgreSQL commit to a queryable ClickHouse row. - 58 dbt tests and 59 unit tests green in CI, and 10 Prometheus alert rules, each with a runbook and each unit-tested with promtool. An audit of that claim found rules with no test at all while promtool still reported success, because it only runs the cases it is given, so the alert names in the tests are now checked against the rules defined. - A convention gate in CI for rules that fail silently when broken: `FINAL` only through the macro, every Replacing engine declaring its sort key, every incremental model declaring its unique key, pinned images, loopback-only ports, and every alert with a duration and a runbook. - Updates and deletes proven to propagate end to end by an executable, not asserted in prose. #### What I would change A Schema Registry with Avro, the one omission with a real correctness consequence. Replicated ClickHouse with Keeper, with the migration path and its trigger volume already written down. Alertmanager routing on the existing rules. Great Expectations at the feature boundary. dbt snapshots over the retained event log to rebuild history, which the current-state marts deliberately do not keep. Airflow runs in a demo topology here; the production topology is described in the design notes rather than half-built. #### Stack and links PostgreSQL, Debezium, Redpanda, ClickHouse, dbt, Apache Airflow, Prometheus, Grafana, Docker Compose, GitHub Actions, Python and Make. The repository, linked below, holds the code, the measured notes on the source API and the CDC wire format, and the CI evidence. ### Kiva loan CDC analytics: PostgreSQL to ClickHouse, orchestrated by Dagster URL: https://matidata.ethioneural.com/work/kiva-microfinance-loan-cdc-analytics Stack: PostgreSQL, Debezium, Redpanda, ClickHouse, dbt, Dagster, Prometheus, Grafana, Docker Compose, GitHub Actions, Python Updated: 2026-10 Note: built as a take-home technical assessment, then extended A take-home technical assessment on Kiva's public loan API, extended since: PostgreSQL, Debezium into Redpanda, ClickHouse with dbt models and tests, Dagster, and Prometheus and Grafana with a reconciliation monitor. In plain words: It copies every change in a loans database into an analytics warehouse as it happens, then compares the row counts on both sides, so a silent gap shows up on a dashboard and in an alert email. #### Context I built this in August 2026 as a take-home technical assessment on Kiva's public loan API, which needs no key or account, and kept improving it through September and October. It is not affiliated with or endorsed by Kiva. A Python job pulls funded loans into PostgreSQL, Debezium streams every change out of the write-ahead log into Redpanda, ClickHouse consumes the topic through its Kafka engine, and dbt builds and tests staging, intermediate and mart models, including a machine-learning feature table. Dagster runs the pipeline, Prometheus and Grafana watch the databases and the stream, GitHub Actions tests the whole path, and `docker compose up -d` starts all of it. I designed it around the question every CDC pipeline is eventually asked: how do you know the connector did not quietly drop events or fall behind, when every service still reports healthy? #### Constraints - One command. `docker compose up -d` brings up every service. A one-shot container renders the Debezium connector config from `.env` and registers it, skipping the step if the connector already exists, and Dagster runs the pipeline once at startup before its schedule takes over. - Laptop-sized. Every service has an explicit memory limit, so exhaustion shows up as a container killed for memory instead of a slow host, and the whole stack needs about 4 GB of free memory. - A free public API. The client sends a browser User-Agent because the API's firewall rejects the default Python client, retries with exponential backoff, and polls every fifteen minutes: often enough to stay current, rarely enough not to hammer a free service. - Everything as code: the connector config, the ClickHouse tables, the dbt models and tests, and Grafana's datasources, dashboards, alert rules, contact point and notification policy. #### Architecture The Kiva loan CDC pipeline: data moves top to bottom, and orchestration and observability run across every stage. Stages, from source to serving: - Kiva public API: Funded loans, no key - Python ingestion: Retry with backoff, upsert on the loan id - PostgreSQL 16: Logical replication for change capture - Debezium 2.5: Reads the write-ahead log through pgoutput - Redpanda: Kafka-compatible event log - ClickHouse: Kafka engine into ReplacingMergeTree - dbt: Staging, intermediate and marts - Grafana: Loan analytics over the marts - Orchestration, across every stage: Dagster: ingestion, then dbt run and dbt test; GitHub Actions CI - Observability, across every stage: Prometheus and Grafana; cdc-monitor: row drift, replication lag, connector state; alert email through Mailpit Ingestion is a small Python client that fetches a few hundred funded loans per run and upserts them into PostgreSQL 16 on the loan id. A re-run updates the status, the funded amount and an `updated_at` timestamp on existing rows, so CDC has real updates to capture, not only inserts. Debezium 2.5 reads the WAL through `pgoutput` and publishes flattened row images to Redpanda as plain JSON, with the operation and a deleted flag on every event. ClickHouse reads the topic through a Kafka engine table, and a materialized view lands each event in a `ReplacingMergeTree` table ordered by loan id and partitioned by month, carrying the source `updated_at` along as a freshness signal. dbt builds a staging view that reads the raw table with `FINAL` and drops deleted rows, an intermediate view that adds a funding tier and a funding percentage, a mart by country and sector for dashboards, and a per-loan feature table with log transforms, one-hot flags, date features and a fully-funded label. The borrower name is tagged as personal data in the dbt metadata. Dagster runs two assets in order, ingestion and then `dbt run` followed by `dbt test`, once at startup and every fifteen minutes after; the CDC path itself does not wait for that schedule. Prometheus scrapes Redpanda and ClickHouse natively, PostgreSQL through postgres-exporter, and a custom `cdc-monitor` exporter that polls both databases and the Kafka Connect API. Grafana is provisioned from the repository with two dashboards, one for pipeline health and one for loan analytics over the marts, plus the alert rules. dbt docs and the Redpanda and Debezium consoles run in the same stack for inspection. #### Decisions and tradeoffs Three decisions I would defend, each aimed at a failure that infrastructure metrics do not show: - Reconcile instead of inferring. Healthy Redpanda and ClickHouse metrics do not prove that every row arrived, so `cdc-monitor` compares the PostgreSQL row count with the deduplicated ClickHouse count, and measures freshness as the age of the newest replicated `updated_at`. A stalled or lossy connector shows up as drift or lag even while every service reports healthy. - No data is not zero lag. Until a row has replicated, the monitor reports no lag at all rather than a reassuring zero, so an empty warehouse cannot pass for a fresh one. - Connector health comes from the Kafka Connect REST API, not from throughput. The connector counts as healthy only when it and every one of its tasks report running, and a connector with no tasks counts as down, because from the outside an idle stream and a dead one look the same. Two more are worth naming. The raw table is a `ReplacingMergeTree` read through `FINAL` in exactly one staging model, so dashboards and features see the latest state of each loan rather than every CDC version, at the cost of merge work at read time that the scaling notes plan around. And monitoring runs inside Compose instead of a hosted service, so nothing needs an external account, while Redpanda stands in for Kafka to keep the broker light enough for a laptop. #### Outcome - 15 dbt tests, unique, not-null and accepted-value checks plus a custom singular test, and 15 unit tests for the ingestion client and the monitor's drift, lag and connector-health logic, all mocked so they run without live services. - 3 CI stages in GitHub Actions. Lint with unit tests, and `docker compose config` with `dbt parse`, run first. Only when both pass does an end-to-end job start PostgreSQL, Redpanda, Debezium and ClickHouse, register the connector, ingest live data from the API, wait for the rows to reach ClickHouse, run `dbt run` and `dbt test` against the warehouse, and check that the monitor reports real metrics. - 4 Grafana alert rules, provisioned as code, for row drift, replication lag, a stopped connector and stalled ingestion, routed through a contact point and a notification policy to email, which Mailpit catches locally. I tested the path with a real fault: stopping the Debezium container flipped the connector metric, the rule went from pending to firing, the alert email arrived, and a resolved email followed after the restart. - A Grafana crash traced to its cause instead of papered over. Once alerting was added, Docker began killing Grafana for running out of memory. The causes were bundled app plugins installed on every boot and a memory limit sized for dashboards alone. My first fix, a global switch, also blocked the one plugin the stack needs, which I caught in the same investigation. A one-shot installer container now adds only the ClickHouse datasource plugin, the default installs stay off and the limit was resized, and soak tests confirmed the fix. - Stable datasource ids. A dashboard had hard-coded a datasource id from an earlier local Grafana session, which would have blanked every panel on a fresh clone; both datasources now have fixed ids that the dashboards and alert rules share. #### What I would change The design notes carry a scaling plan by volume tier. In short: incremental dbt models and a TTL on the raw table once volume grows past demo scale. A schema registry with Avro or Protobuf instead of plain JSON, to stop upstream schema drift before it reaches ClickHouse. Debezium reading from a standby replica rather than the primary, with PgBouncer in front of PostgreSQL. At higher volumes, Kafka with partitioned topics, more connector tasks and a replicated ClickHouse cluster. Feast in front of the feature table for online serving. And with more sources and teams, Airflow with dedicated executors, a dbt project split by domain, and lineage through OpenLineage, because per-table row-count reconciliation stops being enough. #### Stack and links PostgreSQL, Debezium, Redpanda, ClickHouse, dbt, Dagster, Prometheus, Grafana, Docker Compose, GitHub Actions and Python. The repository, linked below, holds the code, a design report with the ClickHouse table-design rationale and the scaling plan, and notes on the data model and observability. The loan data comes from Kiva's public API; the project is not affiliated with or endorsed by Kiva. ### CDC Starter: PostgreSQL to ClickHouse CDC in one command URL: https://matidata.ethioneural.com/work/cdc-starter Stack: PostgreSQL, Debezium, Redpanda, ClickHouse, dbt, Docker Compose, Make, Bash, GitHub Actions Updated: 2026-10 A template of the CDC pattern I use, small enough to read in one sitting: PostgreSQL, Debezium into Redpanda, ClickHouse and dbt in one command, with deletes, ordering, lag and reconciliation handled. In plain words: A small copy of my pipeline pattern that anyone can start with one command: change a few rows in a database, watch the analytics warehouse catch up, and let the checks show that nothing was lost. #### Context My two public CDC pipelines, one for World Bank indicators and one for Kiva loans, carry a lot around the core pattern: an API client, an orchestrator, dashboards and alert rules. When someone asks how to start their own, that is too much to read before the first row moves. CDC Starter is the pattern with everything else taken out, and with the parts that usually go missing put in from the start: deletes, ordering by log position, lag measured against a heartbeat, and a check that the warehouse really matches the source. It is built to be copied. `make demo` starts PostgreSQL, Debezium, Redpanda and ClickHouse, registers the connector, writes inserts, updates and a delete to the source, waits until ClickHouse shows exactly the same orders, builds and tests the dbt models, and runs the checks. It is the running example behind the CDC workshop and the step-by-step build guide on this site. #### Constraints - One command, and safe to run again: a second `make demo` passes like the first. - Laptop-sized. Every image is pinned, every container has a memory cap, about five gigabytes in total, and ports bind to loopback on non-default numbers, so the stack cannot collide with a database already running on the machine. - Nothing to install but Docker, `make` and `curl`. dbt runs in its own pinned image, so nobody needs a local Python environment. - Small enough to read: one table, one connector, two dbt models, and checks written as a shell script with a meaningful exit code instead of a dashboard. #### Architecture The CDC Starter template: data moves top to bottom, and the checks that prove it arrived run across every stage. Stages, from source to serving: - PostgreSQL 17: Explicit publication, a narrow CDC role, a heartbeat table - Debezium 3.6.1: Reads the write-ahead log through pgoutput - Redpanda: Kafka-compatible event log - ClickHouse 25.8: Raw JSON, a typed view, ReplacingMergeTree by LSN - dbt: One current-state macro, then staging and a mart - Quality, across every stage: 10 dbt tests, one that fails if a deleted order returns; row counts and every order's state reconciled - Observability, across every stage: Heartbeat lag under a limit; connector and task state from Kafka Connect The source is an `app.orders` table in PostgreSQL 17 with logical replication on and a trigger that moves `updated_at` on every change. The publication lists only the tables meant for capture, and Debezium connects as its own role, with replication and read access and the one update its heartbeat needs. Debezium 3.6.1 reads the write-ahead log through `pgoutput` and publishes flattened rows to Redpanda. Each event carries the operation, the log position (LSN) and a delete flag, and a delete arrives as a row of its own instead of a bare tombstone. ClickHouse 25.8 reads each message through a Kafka engine table as one raw JSON string. A materialized view types it and writes `raw.orders`, a `ReplacingMergeTree` versioned by the LSN and told which rows are deletes, so the latest change in the source wins rather than the latest to arrive. dbt reads the current state through one macro, `FINAL` plus the delete filter, into a staging view, then builds a mart of orders by status. Source freshness on the raw table catches a stopped pipeline while the mart still looks fine. `make check` then reconciles the source and the warehouse, measures lag against the heartbeat, and asks Kafka Connect whether the connector and its task are running. #### Decisions and tradeoffs - Order by log position, not by arrival. The version column is the PostgreSQL LSN, so a late or redelivered event cannot overwrite a newer one. Deduplicating on an ingestion timestamp looks the same in a demo and goes wrong in production. - Deletes are rows. The connector rewrites a delete as a row with a delete flag, and `ReplacingMergeTree` hides it under `FINAL`. Without this, a delete either vanishes or comes back as a row with empty columns. - The current-state read lives in one macro. Forgetting `FINAL` gives duplicates, and forgetting the delete filter brings deleted rows back. Neither raises an error, so no model writes them by hand, and a singular test fails the build if a deleted order ever reaches staging. - Raw JSON first. The Kafka engine reads a string and the view types it, so a malformed message is skipped instead of stalling the consumer. The cost is that nothing enforces a schema at the edge, which a schema registry would. - An explicit publication and a narrow role. Debezium is told not to create its own publication, so adding an unrelated table to the database never starts capturing it by surprise. - A heartbeat for lag. Debezium updates a heartbeat row on a timer, which measures lag even when the business tables are quiet and keeps the replication slot moving. No heartbeat at all counts as a failure, not as zero lag. - The slot survives restarts. The connector keeps its slot when it stops, so a restart resumes from the recorded position. A cap on the write-ahead log a slot may hold stops a forgotten slot from filling the disk, and `make down` removes everything, the slot included. #### Outcome - `make demo` passes from a clean start, after `make down` has removed every container and volume, and passes again when run a second time. - 4 checks after every run, each printing PASS or FAIL: the connector and its task running according to the Kafka Connect API, row counts that agree, the same state for every order on both sides, and heartbeat lag under a limit. - 10 dbt tests: keys, accepted values and not-null checks on the staging view and the mart, plus the singular test that fails if a deleted order reaches staging. - The failure path is tested, not only the happy one. With the connector paused and nothing written, the row counts still agree: the connector check fails at once, and the heartbeat lag check fails once the limit passes. Changing the source while paused fails the reconciliation as well. After a resume every check passes again and nothing is lost, because the replication slot held the changes. #### What I would change - A make target that runs the pause drill by itself, so CI exercises the failure path and not only the happy one. - The checks exported as Prometheus metrics with alert rules, as in the two larger projects, once the template is used for more than learning. - A schema registry with Avro or Protobuf instead of plain JSON, once more than one team writes to the source. - A second table and a join in the mart, to show how `FINAL` and the delete filter behave across tables. - An orchestrator for dbt, Airflow or Dagster as in the larger projects, once the models need a schedule rather than a command. #### Stack and links PostgreSQL, Debezium, Redpanda, ClickHouse, dbt, Docker Compose, Make, Bash and GitHub Actions. The repository, linked below, holds the code and a README that walks through how data moves, what each check catches, the design choices, and how to adapt the template to your own tables. It is MIT licensed, and a GitHub Actions workflow runs the same `make demo` on a fresh runner for pull requests. The two larger projects it comes from are linked below as well. ### Local RAG assistant with SSO, PII redaction and an audit log URL: https://matidata.ethioneural.com/work/local-rag-assistant Stack: Open WebUI, nginx, FastAPI, Keycloak, AnythingLLM, Ollama, PostgreSQL, Redis, Docker Compose, Python Updated: 2026-10 A document assistant that runs entirely on local hardware: nginx with TLS, a FastAPI gateway with JWT and Keycloak SSO, PII redaction before inference, AnythingLLM retrieval and Ollama, in Amharic and English. #### Context I wanted to know how much of a useful document assistant fits on local hardware with no external service anywhere in the path: the model, the retrieval, the identity provider and the audit log all run on one machine. It answers in Amharic and English. I built it alone on my own laptop GPU, as a personal learning project. The demo runs on sample documents. #### Constraints - No external services: no hosted model API, no hosted vector store, no third-party identity service. - Guardrails in code rather than in policy: raw question text is never stored, personal data is redacted before the model sees it, and every request leaves an audit row. - Role awareness: each of the four roles, from the least privileged to the administrator, sees only its own document set. - Bilingual without two deployments. #### Architecture Open WebUI is the chat interface. nginx terminates TLS and applies per-IP rate limits on the chat and sign-in routes. Behind it, a FastAPI gateway validates the JWT, issued either locally or by Keycloak with role-mapped claims, runs the question through PII redaction, maps the caller's role to an AnythingLLM workspace and forwards the scrubbed text. AnythingLLM retrieves over the documents in that workspace and calls Ollama, which runs on the host GPU outside Docker. PostgreSQL stores users and audit rows, Redis caches sessions, and Keycloak provides single sign-on. Redaction covers phone numbers, account numbers, tax identification numbers (TINs), national ID numbers, birth dates and email addresses. The audit row keeps a one-way hash of the original text, the scrubbed text, the workspace, the role at query time, whether personal data was detected and the response time. The original question is never stored. Generation uses a multilingual instruct model from the Qwen 2.5 family, with a local embedding model for retrieval; both can be swapped in AnythingLLM. The sign-in realm, container names, workspace slugs and certificate subject are rendered from one configuration block by two scripts. #### Decisions and tradeoffs - A gateway in front of the retrieval engine instead of modifying it. Authentication, redaction, role mapping and audit live in one small codebase I own; the cost is an extra hop and configuration in two places. - Redact before inference and log hashes, not text. The model never sees the personal data the patterns catch, and the audit log cannot leak it. The cost is that redaction is pattern-based in this phase: it will miss unusual formats, and it can mangle a legitimate number inside a question. - One workspace per role rather than document-level permissions. Simple to reason about and to audit; coarse, because a document is either in a workspace or not. - Ollama on the host, not in a container, for direct GPU access. One component sits outside Compose and is documented separately. - A small local model over a hosted frontier model. Keeping the data local wins and quality loses, Amharic quality most of all. My research on quantizing Amharic language models is about exactly this trade. #### Outcome A working demo: sign in, ask a question, get an answer grounded in the sample documents, and watch the redaction and audit guardrails fire on a question that contains personal data. The gaps are real, and I would rather state them than let the README imply otherwise: there is no streaming (the request schema has a stream flag, but the gateway returns complete responses), there are no automated tests yet, development runs on self-signed certificates, and redaction is regular expressions only. I publish no usage numbers for it. #### What I would change - Tests before features: redaction unit tests with Amharic and English fixtures, token validation tests, and one integration test through nginx. - Streaming responses through the gateway. - Named-entity redaction for names and addresses alongside the patterns. - Document-level permissions on top of the role workspaces. - A small Amharic evaluation set, so a model swap is measured rather than felt. - Metrics for Prometheus next to the existing health endpoint. #### Stack and links Open WebUI, nginx, FastAPI, Keycloak, AnythingLLM, Ollama, PostgreSQL, Redis, Docker Compose and Python. The source is private. ### Amharic OCR for Ethiopic script with HHD-Ethiopic and MMOCR URL: https://matidata.ethioneural.com/work/amharic-ocr-ethiopic-script Stack: FastAPI, TensorFlow, OpenCV, PyMuPDF, Next.js, TypeScript, Tailwind CSS, MMOCR, PyTorch, ONNX, Python Updated: 2026-10 A full-stack OCR app for Ethiopic script: hierarchical detection, HHD-Ethiopic CTC recognition with confidence scores, multi-page PDFs and a Next.js canvas, plus an MMOCR research line with SATRN and DBNet++. #### Context Ethiopic script, used for Amharic, Tigrinya and Ge'ez, is poorly served by commercial OCR. Archives, records and teaching material sit on paper or in scanned images that no search engine can read. I built an OCR system for Ethiopic script in two parts: a working full-stack application on top of the pre-trained HHD-Ethiopic recognizer, and a research line for training better detection and recognition models with MMOCR. #### Constraints - Ethiopic is a syllabary with a large character set and many visually close glyphs. The application's recognizer covers 306 Ethiopic characters: the syllabary, punctuation and numerals. - Documents arrive as photos, scans and multi-page PDFs, with skew, uneven lighting and mixed quality. - People need to see what the model saw: region-level results with a confidence for each, not a block of text to take on trust. - Personal hardware and public data only. #### Architecture The backend is a FastAPI service with endpoints for image OCR, base64 input, batch processing, detection only, detect-and-recognize, recognition of a chosen region, and PDF information and OCR. The pipeline has four stages. Preprocessing converts to grayscale, binarises with adaptive thresholding and deskews. Hierarchical detection finds lines with horizontal projection profiles, words with vertical projection and characters with connected-component analysis, returning bounding boxes at all three levels. Recognition feeds each region through the HHD-Ethiopic TensorFlow model and decodes the output with CTC, producing text and a confidence score per region. PDFs are rendered page by page with PyMuPDF and processed as images. The frontend is a Next.js application with a detection canvas that overlays the boxes on the image; clicking a box shows its recognised text, and a results panel shows the line, word and character tree. Upload is drag and drop. The research line uses MMOCR: a SATRN recognizer over a 356-character Ethiopic vocabulary, DBNet++ for detection on Amharic documents, a synthetic data generator for Amharic word images, and ONNX export so trained models can be served without the training stack. #### Decisions and tradeoffs - Ship on a pre-trained recognizer first. The application exists and can be tried today. The cost is domain: HHD-Ethiopic comes from historical handwriting, so modern printed documents differ from what it learned and results vary with the input. - Projection-profile detection in the application rather than a learned detector. No training, fast, and every box is explainable. It struggles with strong skew, multi-column layouts and dense handwriting, which is why DBNet++ is in the research line. - Confidence exposed per region. It lets a reviewer decide what to trust. CTC confidences are not calibrated, so a score is a ranking signal, not a probability. - Two repositories. Training dependencies (MMOCR, PyTorch) and serving dependencies (TensorFlow, FastAPI) do not belong in one environment, and ONNX is the bridge between them. #### Outcome A working application: upload an image or a PDF, see detected lines, words and characters on the canvas, read the recognised text with its confidence, and call the same pipeline through the API, including in batch. The research line has training configurations for the recognizer and the detector, a synthetic data generator and an export path. No accuracy figures are published yet: the research README states a target error rate, and a target is not a result. #### What I would change - Evaluate first: character and word error rates on a held-out set of real documents, published with the split. - Replace projection detection with DBNet++ in the application, and move the recognizer to SATRN through ONNX once both are evaluated. - Fine-tune for modern printed text, where most practical demand is, using the synthetic generator for fonts. - Package the backend with the model fetched at build time, so the demo runs with one command. #### Stack and links FastAPI, TensorFlow, OpenCV and PyMuPDF on the backend; Next.js, TypeScript and Tailwind CSS on the frontend; MMOCR, PyTorch and ONNX in the research line. The application repository is private for now. ### Natural language to SQL analytics agent over ClickHouse URL: https://matidata.ethioneural.com/work/nl-to-sql-analytics-agent Stack: FastAPI, LangChain, Groq, ClickHouse, React, TypeScript, Redux Toolkit, Chart.js, Tailwind CSS, Vite, Vercel, Python Updated: 2026-10 Note: built as a technical assessment prototype A prototype that turns plain-English questions into ClickHouse SQL with a Groq-hosted model and returns tables, charts and a short explanation in a React chat interface; built as a technical assessment prototype. #### Context The brief: let a marketing team get answers from their analytics database without writing SQL. How do users split across segments, which products sell most, how did orders move last month. I built a prototype that translates a plain-English question into ClickHouse SQL, runs it, and returns a table, a chart and a short explanation in a chat interface. It was built over a few days as a technical assessment prototype. It demonstrates an approach; it is not a product, and I claim no usage or impact numbers for it. #### Constraints - A hosted model with fast responses, so the chat feels like a conversation; Groq serves the model. - ClickHouse's SQL dialect, which differs from the SQL most models learned: correlated subqueries are not supported, and a trailing semicolon breaks the client. - Answers readable by people who do not read SQL: a table, a chart where one helps, and a sentence of explanation. - Hosting on Vercel: the backend as a serverless function, the frontend as a static site. #### Architecture The frontend is a React chat interface with Redux Toolkit for state, Chart.js for charts and Tailwind for styling, built with Vite. It posts the question and a session id to a FastAPI endpoint. Small talk and introductions are answered without touching the database. For a data question, the agent builds a prompt from the table schemas and the dialect rules, asks the Groq-hosted model for SQL, extracts the statement from the reply and runs it against ClickHouse. If the model reaches for a correlated subquery, the agent asks once more for a rewrite with a join. The result becomes a markdown table, and a chart type is chosen by rules on the shape of the result: a single value becomes a headline figure, one column a table, a category with a measure a bar chart, a date with a measure a line chart. A final model call writes the explanation from the rows. A loader script fills ClickHouse with a sample dataset of users, orders, products and group leaders. #### Decisions and tradeoffs - Rule-based chart selection instead of asking the model which chart to draw. Deterministic and testable, at the cost of being coarse. - Dialect rules in the prompt, plus one targeted retry, rather than a SQL rewriter. Fast to build; the rules hold only as well as the model follows them. - Session memory kept in process. Each session gets a short window of recent turns, but in the repository code the window is saved after every answer and never read back into the prompts, so follow-up questions do not yet see earlier turns. On a serverless backend, in-process memory would also vanish between cold starts. - The guardrail gap, stated plainly. The generated SQL runs as it comes back, under the configured database user. Nothing in the code restricts it to reads: the extractor accepts INSERT, UPDATE and DELETE as readily as SELECT, and the prompt does not ask for read-only queries either. On a real database this is the first thing to fix: a read-only user, a parser-based allowlist of SELECT and WITH statements, a row limit and a statement timeout. I am flagging it here rather than hiding it. #### Outcome The repository's sample questions show the intended scope: segment breakdowns, top products by revenue, orders over time and average order value by segment, each meant to come back as a table, a chart where the rules call for one, and a short explanation. Within that scope the prototype does what the brief asked; the two gaps above mark where it stops. #### What I would change - Read-only enforcement in the database and in the application, as above. - Feed the session window into the SQL prompt, and keep it in Redis or PostgreSQL so it survives cold starts. - Schema-aware retrieval of table and column descriptions, so the prompt scales past a handful of tables. - A validation loop: parse the SQL, run it with a limit, and return errors to the model once. - An evaluation set of questions with golden SQL, so prompt changes are measured. #### Stack and links FastAPI, LangChain and Groq on the backend, ClickHouse for storage, and React, TypeScript, Redux Toolkit, Chart.js, Tailwind CSS and Vite on the frontend, hosted on Vercel. The source repository is not public at the time of writing. ### Hair virtual try-on: a React and FastAPI prototype URL: https://matidata.ethioneural.com/work/hair-virtual-try-on Stack: React, TypeScript, Vite, Tailwind CSS, Framer Motion, FastAPI, Python, Pillow, Docker, Hugging Face Spaces, Vercel Updated: 2026-10 Note: the model step is a placeholder blend; HairFastGAN is not integrated yet A full-stack prototype: a React and TypeScript app to upload a photo and a hairstyle reference, compare before and after, and download the result, over a FastAPI endpoint whose model step is still a placeholder. #### What it was A two-day prototype from June 2025 for a virtual hair try-on. The front end, in React and TypeScript with Vite, Tailwind CSS and Framer Motion, takes your photo, a photo with the hairstyle you want and an optional colour reference by drag and drop, shows a processing screen, and returns the result beside the original with a before-and-after slider and a download button. Behind it is a FastAPI endpoint with input validation and a health check, packaged as a Docker image for Hugging Face Spaces. The model step is a placeholder: it resizes the two photos and blends them, so the output is not a real hairstyle transfer. HairFastGAN, the open research model the API is shaped around, is not wired in yet. The front end is deployed on Vercel; the API Space is offline, so the live front end cannot process images. #### What I learned Shaping the API around the model's interface paid off: the endpoint takes the same three images as HairFastGAN's swap step, a face, a hairstyle shape and a colour reference, so the real model can replace the placeholder without changes to the front end. What is missing is the model itself: its pretrained weights and the compute to serve them. ### How small, how quantized? Tokenizer, size and quantization for Amharic SLMs URL: https://matidata.ethioneural.com/work/amharic-slm-quantization Stack: Ollama, GGUF, bitsandbytes, Hugging Face Transformers, QLoRA, PyTorch, Python, MasakhaNEWS, AfriSenti Updated: 2026-10 An independent study of how tokenizer choice, model size and quantization each affect Amharic small language models on a single consumer GPU, with QLoRA recovery tested; submitted to a 2026 workshop, in review. #### Context Running a language model for Amharic in practice means a small model, heavy quantization and consumer hardware. Data residency, cost and unreliable connectivity push practitioners that way, and it is also the corner of the design space with the least published evidence. Two effects are established separately: small models quantize differently from large ones, and non-Latin scripts suffer more under quantization, although that multilingual claim is contested by a study that found no disproportionate harm on a large model. Nobody had measured the compound for Amharic. So I asked a practitioner's question. On one consumer GPU, how small a model and how aggressive a quantization can I choose for an Amharic task before accuracy falls off a cliff, what explains the drop, and does cheap fine-tuning buy the loss back? This is independent research on public datasets only, not affiliated with any employer. The paper, "How small, how quantized? Tokenizer, size and quantization for Amharic SLMs on consumer GPUs", is submitted to a 2026 workshop, in review. #### Constraints - The hardware is the subject, not a proxy for it: everything runs on a single 4 GB laptop GPU. - Public benchmarks only: MasakhaNEWS for topic classification and AfriSenti for sentiment, each paired with an English counterpart so that the script, not the instruction, is the variable. - Statistics first: fixed seeds, deterministic decoding, bootstrap confidence intervals, identical prompts across languages, and reruns verified byte-identical. - Connectivity. Full-precision weights beyond a modest size would not download reliably, which limited the controlled arm and the fine-tuning arm to the smallest model. That limit is itself part of the regime under study. #### Architecture The experimental design is the architecture. The matrix covers instruction-tuned checkpoints from four tokenizer families, Qwen 2.5, Gemma 2, Llama 3.2 and Phi-4, with models up to 8B parameters, plus a size ladder inside one family so that scale can be read with the tokenizer held fixed. Precision has two arms. The deployment-realistic arm runs GGUF `q8_0`, `q4_K_M`, `q3_K_M` and `q2_K` through Ollama, which is what practitioners actually run. The controlled arm quantizes identical weights with bitsandbytes to `fp16`, `int8` and `nf4`, to check the GGUF result with an independent method. Two scoped tasks, each paired Amharic and English, give a chance floor that makes degradation readable. The mechanism variable is tokenizer fertility: tokens per word on paired headlines, Amharic against English, per tokenizer. Recovery is QLoRA on the 4-bit smallest model with the Amharic sentiment training split, evaluated on the disjoint test split. A resumable harness writes one row per model, precision, task and language cell, and every table and figure regenerates from those files. #### Decisions and tradeoffs - Scoped classification tasks instead of broad benchmarks. They are the tasks people deploy, they are cheap to run on a small GPU, and their chance floor makes a drop legible. The cost is that the study says nothing about generation or reasoning. - GGUF through Ollama as the primary arm. It measures what gets deployed, but a quirk of one quantization library could pass for a language effect, which is why the bitsandbytes arm exists. - English as a paired control with the same instruction. It isolates the script effect. Label sets differ by language on the topic task, which understates the effect rather than inflating it. - Measuring fertility rather than citing it. The token premium for non-Latin scripts is known; turning it into a per-model predictor of task accuracy is the contribution. #### Outcome The ordering the paper argues is tokenizer first, then model size, then quantization. The first-order driver of Amharic accuracy is the tokenizer: Ge'ez-script fertility varies several-fold across families, and on the topic task it tracks baseline Amharic accuracy almost perfectly across four families, while English accuracy stays flat. A smaller model with a better Ge'ez tokenizer beat larger models with worse ones. Size helps, then saturates early. Quantization is secondary. In the clearest case, a mid-sized Qwen 2.5 model on Amharic topic classification, 4-bit costs about 3 points, 3-bit about 20 or more, while English degrades gently; a second family shows the same extra Amharic penalty at 3-bit, more mildly. At 2-bit both languages collapse into non-label output, a model break rather than a script effect. The controlled arm confirms, on identical weights, that 4-bit is benign. QLoRA at the smallest scale did not recover the gap: the tuned model collapsed to predicting a single label. My reading is that the tokenizer had already shredded the Amharic input into fragments a model that small could not use. You cannot fine-tune past a tokenizer that destroys the signal. The guidance follows: choose the tokenizer first, do not quantize Amharic below 4-bit, and prefer a smaller well-tokenized model over a larger poorly-tokenized one. #### What I would change - Run the recovery experiment at larger sizes once the weights can be transferred; whether a mid-sized QLoRA closes the Amharic gap is the open question. - Add tokenizer families to tighten the correlation; the sentiment result points the same way but is underpowered. - Extend to generation and reasoning tasks, and to other Ethiopian languages and scripts. - Add human evaluation, since prior work shows automatic metrics underestimate how much degradation people notice. #### Stack and links Ollama and GGUF for the deployment arm, bitsandbytes and Hugging Face Transformers for the controlled arm, QLoRA for recovery, PyTorch and Python, with MasakhaNEWS and AfriSenti as the public tasks. The repository stays private while the paper is in review; both will be linked once review ends. ### Frontend of an HR and payroll ERP at HST Consulting URL: https://matidata.ethioneural.com/work/hst-payroll-frontend Stack: JavaScript, HTML, REST APIs, Git Updated: 2026-10 My first industry role, where I built and maintained the frontend of a software-as-a-service HR and payroll system and integrated it with the backend team's REST APIs. #### What it was My first industry role after graduating, as a software developer at HST Consulting in Addis Ababa. I worked on the frontend of HST Payroll, a software-as-a-service HR and payroll product: the screens HR staff use day to day, form validation, and the calls into the REST APIs the backend team exposed. The work was JavaScript and HTML on the client side, Git for collaboration, and a lot of back and forth with backend developers about API contracts and error handling. It is a commercial product, so there is no public code to link here. #### What I learned Payroll is where small frontend mistakes become money mistakes, so I learned to treat validation and state as correctness problems rather than polish. Working against someone else's API taught me to read contracts carefully and to ask for them in writing, the same habit I later brought to data contracts between pipelines and their consumers. It also showed me that I cared more about the data behind the screens than the screens themselves, which pointed me towards data engineering. ### Student dropout prediction with a Streamlit dashboard URL: https://matidata.ethioneural.com/work/student-dropout-prediction Stack: Python, pandas, scikit-learn, Streamlit, Jupyter Updated: 2026-10 A two-week data science exercise that cleaned and explored a student records dataset, tested hypotheses about who drops out, and shipped the findings as an interactive Streamlit app. #### What it was A short structured exercise from September 2024: predict which students are at risk of dropping out, and make the analysis usable by someone who does not read notebooks. The first week was data wrangling, handling missing values, outliers, types, encoding and derived features, then descriptive statistics, correlation analysis and hypothesis tests on questions such as whether admission grades or financial aid relate to dropout. The second week was exploration and visualisation, from univariate plots to principal component analysis, and a Streamlit dashboard that summarises the insights and lets a user explore the data and the risk predictions. The app runs on Streamlit Community Cloud and I wrote the project up on Medium. #### What I learned Hypothesis tests kept me honest: several patterns that looked obvious in a chart did not survive a t-test or a chi-square test, so I learned to lead with the question and let the plots serve it. Building the dashboard taught me that the last mile, deployment and a clear interface, is what turns an analysis into something an administrator would act on. I also started writing about my work in public, which I have kept doing since. ### Analytics and machine learning project set from 10 Academy URL: https://matidata.ethioneural.com/work/ten-academy-analytics-projects Stack: Python, pandas, scikit-learn, Jupyter, Streamlit, PostgreSQL, dbt, DVC, Redash Updated: 2026-10 A set of weekly end-to-end projects from the 10 Academy training programme, covering forecasting, fraud detection, credit scoring, a Telegram-sourced data warehouse and several exploratory analyses. #### What it was Between April and July 2024 I worked through the 10 Academy programme: one real-world project a week, each ending in a report. The set covered sales forecasting on the Rossmann store data, time-series analysis of Brent oil prices, fraud detection on e-commerce transactions, credit scoring built from e-commerce transaction data, a data warehouse for medical business data scraped from Telegram channels, insurance claims analysis with hypothesis testing and a predictive model, a marketing analytics dashboard, exploratory analysis of solar farm data in Streamlit, and a telecom customer analysis. Most weeks ended with a Medium write-up, and the warehouse week is where I first used dbt and a proper ELT layout. #### What I learned Repetition under a deadline built habits: start from the business question, profile the data before modelling, keep the pipeline reproducible and write the report as you go. The fraud and credit scoring weeks taught me how much class imbalance and leakage shape a model's apparent quality. The warehouse week mattered most in hindsight: scraping, loading, transforming with dbt and serving a dashboard was a small version of the work I now do in production. ### Cotton disease prediction with deep learning (final-year project) URL: https://matidata.ethioneural.com/work/cotton-disease-prediction-deep-learning Stack: TensorFlow, Keras, Flask, TensorFlow Lite, Android Updated: 2026-10 My final-year project classified cotton leaf diseases from photos with a convolutional network trained by transfer learning, served through a Flask web app and an Android app for use in the field. #### What it was My final-year project for the BSc in Electrical and Computer Engineering at Wolaita Sodo University. Cotton growers had no quick way to tell leaf diseases apart in the field, so I trained an image classifier on labelled cotton leaf photos using transfer learning on a pretrained ResNet in TensorFlow and Keras. The model served two front ends: a Flask web app where a user uploads a photo and gets the predicted disease class, and an Android app that runs a TensorFlow Lite export of the same model on the phone, so a diagnosis does not depend on a connection. The thesis, the training notebook and both apps are in the repos linked below. #### What I learned Transfer learning made a small dataset usable, but the gap between notebook results and field photos taught me more than the model did: lighting, background and phone cameras all shift the input. Exporting quantised and unquantised versions for the phone forced me to think about model size and precision long before I met the same tradeoffs in production. It was also the first time I shipped one model through two interfaces, a pattern I still use. ### Library database system in MySQL (database course project) URL: https://matidata.ethioneural.com/work/library-database-system Stack: MySQL, MySQL Workbench, SQL Updated: 2026-10 A course project that modelled a university library in MySQL, from the extended entity-relationship design through to the relational schema and the SQL scripts behind it. #### What it was A group project for the database systems course in my degree. We modelled the things a university library has to track, drew the extended entity-relationship diagram in MySQL Workbench, turned it into a relational schema, and wrote the SQL that creates the tables, loads sample data and answers the everyday questions a librarian asks of the system. The repo holds the Workbench models, the diagrams, the SQL scripts and the written report we handed in. #### What I learned This was where SQL stopped being syntax and became modelling. Deciding what a row means, where a foreign key belongs and which constraints protect the data was the real work; the queries followed from a good schema. I still start every warehouse or lakehouse design with the same questions about grain and keys, only now with many more tables and with the diagram living in dbt rather than in Workbench. ### WSU Guide, a campus navigation app for Wolaita Sodo University URL: https://matidata.ethioneural.com/work/wsu-guide-android-app Stack: Android, Java, Gradle Updated: 2026-10 A native Android app that helps new students, visitors and staff find offices, dormitories, departments and services across the two Wolaita Sodo University campuses. #### What it was My first shipped software, built while I was a student. New students, guests and even staff at Wolaita Sodo University kept getting lost between the main campus and the technology campus, so I built an Android app that lists offices, dormitories, departments, the registrar and other services and guides the user to them from their current location. It covers both campuses. I built it as a native Android project, designed the screens myself and recorded a walkthrough video so people on campus could see what it did before installing it. #### What I learned Taking a project from an idea to something other people installed was the lesson. I learned that the data behind an app, every room and office name, needs maintaining as much as the code does, and that a mobile product is mostly design and testing rather than programming. It also gave me the confidence to take on the deep learning project the following year, and it is why the years sentence on this site starts in 2021.