Lead I - Data Engineering

5 - 7 Years
2 Openings
Pune

Job Title: Senior Data Engineer – Azure & Databricks

Overview

We are seeking a highly skilled Senior Data Engineer with 5+ years of hands-on experience in building scalable, high-performance data solutions within the Azure ecosystem. The ideal candidate will have strong expertise in Databricks, Apache Spark (Java), modern data lake architectures, and end-to-end ETL pipeline development. This role involves designing, developing, and optimizing data systems that support business-critical analytics and operational use cases.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure services and Databricks.

• Build efficient data ingestion and transformation workflows leveraging Databricks, Spark, and Delta Lake.

• Develop, enhance, and optimize ETL pipelines using Java, Spark, and SQL.

• Implement industry-best practices for data modeling, data quality, governance, and pipeline performance.

• Work with relational and lakehouse data platforms, including PostgreSQL and Delta Lake.

• Collaborate with cross-functional teams to ensure smooth execution, deployment, and support of production data workloads.

• Troubleshoot performance issues and optimize workloads for reliability and scalability.

Required Skills & Experience

• 5+ years of hands-on experience in Data Engineering.

• Strong working knowledge of the Azure ecosystem, specifically:

  • Azure Databricks

  • Azure Data Lake Storage (ADLS)

  • Azure Functions

  • Azure Data Factory (preferred but not mandatory)

• Expertise in Apache Spark with Java (Spark Core, Spark SQL, Spark Streaming).

• Advanced SQL skills including performance tuning and complex query optimization.

• Strong understanding of ETL/ELT methodologies, data warehousing concepts, and lakehouse architectures.

• Experience with Delta Lake features such as schema evolution, ACID transactions, and time travel.

• Hands-on experience with PostgreSQL or similar relational databases.

• Excellent analytical, problem-solving, and debugging skills.

• Strong communication skills with the ability to work with technical and non-technical stakeholders.

• Self-motivated, proactive, and comfortable working in fast-paced environments.

Preferred Skills

• Experience with CI/CD tools such as Azure DevOps and Git.

• Knowledge of Infrastructure-as-Code (IaC) tools like Terraform.

• Exposure to streaming technologies such as Kafka or Azure Event Hub.

• Familiarity with data governance tools such as Azure Purview.

Azure Cloud,Java,Pyspark