Principal Data Engineer (Databricks / PySpark, DBT & Astronomer / Airflow)

We are seeking a Principal Data Engineer with 5+ years of hands-on experience in the modern cloud data stack. This role is ideal for someone passionate about building robust, scalable data solutions and who thrives in agile, collaborative environments.

The core technical stack centers around Databricks / PySpark for large-scale data processing, DBT for transformation, Astronomer / Airflow for orchestration (with Cosmos to run DBT DAGs natively in Airflow), and Snowflake as the cloud data warehouse.

Key Responsibilities

• Build and maintain scalable data pipelines using DBT and Astronomer/Airflow for batch and incremental data processing.
• Develop modular DBT transformation layers (staging intermediate mart) with robust testing, documentation, and incremental model strategies.
• Orchestrate end-to-end workflows on Astronomer / Airflow, using Cosmos to trigger and manage DBT models as native Airflow DAGs.
• Design and optimize Snowflake schemas, queries, and cost controls for analytical workloads.
• Enforce data quality standards through DBT tests and pipeline observability tooling.
• Write clean, well-tested Python and SQL code and actively participate in code reviews.
• Lead and mentor a team of data engineers; contribute to sprint planning, architectural decisions, and technical standards.

Required Skills & Experience

• Databricks & PySpark: Delta Lake, batch and streaming workloads, cluster and cost optimization.
• DBT (Core & Cloud): Advanced data modelling patterns, incremental strategies, custom tests, and CI/CD integration.
• Astronomer / Airflow: DAG authoring, Cosmos for DBT orchestration, and pipeline observability.
• Snowflake: Data modelling, query performance tuning, access control, and cost governance.
• Python and SQL: Strong programming skills with a focus on modular, maintainable, and testable code.
• Cloud platform: Hands-on experience with Azure or AWS.
• Agile & CI/CD: Comfortable working in agile teams with a CI/CD development mindset.
• Degree: Bachelor’s in Computer Science or a related discipline.

We have an amazing team of 700+ individuals working on highly innovative enterprise projects & products. Our customer base includes Fortune 100 retail and CPG companies, leading store chains, fast-growth fintech, and multiple Silicon Valley startups.

What makes Confiz stand out is our focus on processes and culture. Confiz is ISO 9001:2015 (QMS), ISO 27001:2022 (ISMS), ISO 20000-1:2018 (ITSM), ISO 14001:2015 (EMS), ISO 45001:2018 (OHSMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.

People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves.

To know more about Confiz Limited, visit: https://www.linkedin.com/company/confiz-pakistan/