Data Engineer

Data Engineer (6–7 Years Experience)

Location: Remote (Required to visit client office 1–2 days/month – any location)
Working Hours: 11:00 AM – 9:00 PM IST

Role Overview

We are looking for an experienced Data Engineer with 6–7 years of hands-on experience in building data pipelines, working with Azure Databricks, ADF, Python, Apache Spark, and SQL. The candidate will collaborate with business and technical teams to develop scalable data solutions and support analytics initiatives.

Key Responsibilities

• Design and develop scalable data pipelines using Azure Databricks, Apache Spark, and related Azure services.
• Create and manage workflows using Azure Data Factory (ADF) and automation platforms.
• Integrate structured, semi-structured, and unstructured data into Databricks environments.
• Develop and maintain ETL/ELT pipelines using Delta Lake and Azure Data Lake.
• Optimize Spark jobs for performance, reliability, and cost efficiency.
• Troubleshoot and resolve issues in distributed data workflows.
• Collaborate with data analysts, business users, and teams for self-service analytics support.
• Automate data ingestion, transformations, deployments, and workflow orchestration.
• Implement CI/CD pipelines using GitHub Actions or similar tools.
• Ensure adherence to data governance, security, and privacy requirements.
• Utilize data cataloging and lineage tools to maintain metadata governance.

Required Skills (Must-Have)

• Azure Databricks
• Azure Data Factory (ADF)
• Python
• Apache Spark
• SQL

Good-to-Have Skills

• Power Apps
• Power Automate
• Azure Logic Apps
• ETL process knowledge
• GitHub Actions / CI-CD pipelines
• MLflow
• Machine learning workflows

Experience Requirements

• 6–7 years of experience in data engineering.
• Strong understanding of ETL/ELT methodologies and data modeling.
• Hands-on experience with Azure ecosystem: Databricks, Data Lake, Synapse, and ADF.

Preferred Qualifications

• Experience building CI/CD pipelines for Databricks.
• Knowledge of MLflow, Delta Lake, and machine learning lifecycle workflows.
• Familiarity with Azure Purview or other governance platforms.
• Experience working in Agile environments.

Soft Skills

• Strong communication and stakeholder management capabilities.
• Ability to work in fast-paced, dynamic environments.
• Excellent problem-solving skills and attention to detail.

Job Type: Contractual / Temporary
Contract length: 6-12 months

Pay: ₹70,000.00 - ₹90,000.00 per month

Work Location: Remote