Manager - Data Engineering

Job Title: Principal Data & AI Architect / Databricks Solutions Architect

Location: [Various - e.g., Hybrid, Bengaluru, Chennai, Remote]

Experience Required: 8+ to 15+ years (Senior/Principal level)

Experience in AI/ML development, a strong foundation in data-driven modeling, and a passion for applying advanced analytics to real-world problems. You will collaborate closely with engineers, data scientists, and product managers to develop innovative solutions that provide measurable impact.

Architecture & Design: Define end-to-end Lakehouse architectures, including Medallion architecture (Bronze, Silver, Gold), Delta Lake, and Unity Catalog for governance.

AI/ML Integration: Implement production-grade AI/ML pipelines, including Feature Engineering, Model Training, and Deployment using Databricks MLflow and Databricks Jobs/Workflows.

Generative AI & LLMs: Develop Retrieval-Augmented Generation (RAG) applications, fine-tuning LLMs, and implementing LLMOps workflows.

Data Engineering & Optimization: Optimize performance, cost, and reliability of large-scale Spark workloads and streaming data pipelines.

Migration Strategies: Lead data migration from legacy systems (e.g., Snowflake, Hadoop, Teradata) to the Databricks Lakehouse.

Governance & Security: Define security models (RBAC, row/column-level security) and data lineage using Unity Catalog.

Consulting & Leadership: Act as a technical advisor to stakeholders, mentoring junior engineers, and conducting Proof-of-Concepts (PoCs).

Databricks Artchitect

AI

Chennai, Tamil Nadu, India