DE&A - Core - DataBricks Data Architect

We are seeking an experienced Data Architect with deep expertise in Databricks and modern data lakehouse architectures across cloud platforms such as Azure, AWS, or GCP . The ideal candidate will be responsible for designing, governing, and leading large-scale data modernization and analytics initiatives. This role involves architecting data platforms that enable scalable data engineering, analytics, and AI/ML workloads using Databricks as the core platform.

Define and implement end-to-end data architecture using Databricks (Delta Lake, Unity Catalog, Spark, MLflow).

Design data ingestion, transformation, and storage frameworks supporting structured, semi-structured, and streaming data.

Architect data lakehouse solutions for enterprise-scale analytics and AI workloads.

Collaborate with stakeholders to translate business, analytics, and ML requirements into scalable data platform designs .

Develop standards and best practices for Databricks clusters, job orchestration, and optimization.

Work with data engineers, data scientists, and platform teams to ensure seamless integration and scalability.

Establish governance, security, and data lineage using Databricks Unity Catalog or third-party tools.

Define and implement data modeling , metadata management , and data quality frameworks .

Support multi-cloud deployment strategies (Azure, AWS, or GCP) and hybrid data environments.

Lead efforts on performance tuning, cost optimization, and DevOps integration for Databricks workloads.

Provide technical leadership and mentorship to offshore engineering teams.

12+ years of overall experience in data architecture, engineering, and analytics platforms .

Strong expertise with Databricks , Delta Lake , Apache Spark , PySpark , and MLflow .

Hands-on experience with at least one major cloud platform ( Azure , AWS , or GCP ) and its native data services:

Azure: Data Lake Storage, Synapse, ADF

AWS: S3, Glue, Redshift, EMR

GCP: BigQuery, Dataflow, Dataproc

Proven experience designing lakehouse and ELT frameworks for large-scale enterprise data.

Strong understanding of data modeling, schema design, partitioning , and data lifecycle management .

Expertise in Python , SQL , or Scala for data processing and orchestration.

Experience implementing CI/CD pipelines , GitOps , and infrastructure automation (Terraform, CloudFormation).

Knowledge of data governance, cataloging, and access control frameworks (Unity Catalog, Purview, Glue Data Catalog).

Strong communication and architecture documentation skills.

Experience with real-time streaming (Kafka, Event Hubs, Kinesis).

Exposure to MLOps or AI/ML lifecycle management in Databricks.

Familiarity with Snowflake , dbt , or modern ELT tools .

Cloud certifications such as:

Databricks Certified Data Engineer / Architect Professional

AWS Solutions Architect / Azure Architect Expert / GCP Data Engineer

Bachelor’s or Master’s degree in Computer Science, Information Systems, or Data Engineering .

Advanced cloud or Databricks certifications preferred.