Transit Lakehouse
Scheduled micro-batch lakehouse capturing real-time transit telemetry into Apache Iceberg on Amazon S3 Tables. Orchestrated with Airflow 3 on AWS Fargate, transformed with dbt-athena, and provisioned with Terraform.
Deep dive & architecture
Open table format with Apache Iceberg: Stores transit telemetry in Amazon S3 Tables backed by the AWS Glue catalog. Demonstrates schema evolution across changing GTFS stop specifications, managed snapshot compaction, and metadata-driven partition pruning without maintenance decay.
Disciplined 60s micro-batching: Uses EventBridge and AWS Lambda to ingest raw protobuf GTFS-Realtime feeds every minute into immutable S3 partitions. A lightweight PyIceberg writer batches records directly into Iceberg tables, avoiding the operational bloat and cost of always-on streaming consumers.
Airflow 3 asset orchestration: Deploys Apache Airflow 3 on AWS Fargate using LocalExecutor, leveraging native asset-aware scheduling and blocking data freshness checks to trigger dbt-athena modeling runs only when new partitions land.
Targeting real feed pathologies: Profiles transit feed quirks before modeling: handling mid-trip vehicle signal dropouts, out-of-order telemetry coordinates, and real-time trips unlisted in static schedules.
Infrastructure as Code & cost invariants: Entire environment is provisioned via Terraform with a verified destroy path and zero hardcoded credentials (OIDC CI). Strict Athena workgroup byte-scan caps, S3 lifecycle expiration, and budget alarms act as asserted invariants to maintain a steady-state run cost under ~$13/month.