Data Engineering

Pipelines, warehouses and the modelling that makes analytics trustworthy.

1. Step 01

   SQL mastery

   Window functions, CTEs, query planning and performance tuning on large tables.

   SQLPostgreSQL

2. Step 02

   Programming

   Python for ETL, file formats, and writing idempotent, restartable jobs.

   PythonParquet

3. Step 03

   Warehousing

   Dimensional modelling, star schemas, slowly changing dimensions and partitioning.

   BigQuerySnowflake

4. Step 04

   Orchestration

   DAGs, dependencies, backfills, retries and data-quality tests as first-class steps.

   Airflowdbt

5. Step 05

   Streaming

   Event logs, exactly-once semantics, windowing and late-arriving data.

   KafkaFlink

6. Step 06

   Governance

   Lineage, cataloguing, PII handling, cost control and access policies.

   LineageGovernance