← projects

Incremental Customer Master Data Pipeline

PySpark · SQL · SCD Type 2

  • Analyzed customer data changes using PySpark and hash-based change detection to identify newly inserted and modified records.
  • Validated historical customer records using Slowly Changing Dimension Type 2 logic and SQL reconciliation checks.
  • Investigated discrepancies across staging, intermediate, and warehouse layers to ensure consistent customer information.