Senior Data Engineer – Databricks

Senior Data Engineer – Databricks

Location: Hyderabad/Chennai

Work Schedule: Timings 2:30 PM to 11:30 PM

Role Overview:

We are seeking a highly skilled Senior Data Engineer with deep expertise in Databricks to design, build, and optimize scalable data pipelines and architectures. You will collaborate with cross-functional teams including data scientists, analysts, and platform engineers to deliver high-quality, reliable, and secure data solutions.

Key Responsibilities:

• Design, develop, and maintain robust batch and streaming data pipelines using Databricks, Apache Spark, Delta Lake, and PySpark.
• Build and optimize ETL/ELT workflows to transform and integrate data from diverse sources.
• Implement Delta Live Tables (DLT) and Databricks Workflows for pipeline orchestration and automation.
• Ensure data quality, observability, and monitoring across all pipelines.
• Collaborate with cloud platform teams (AWS/Azure) to ensure proper setup of storage, networking, and security.
• Work within Agile teams, participating in sprint planning, code reviews, and deployments.
• Document pipeline processes, technical specifications, and architectural decisions.
• Stay current with emerging technologies and best practices in data engineering and cloud-native solutions.

Required Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• Around 5 years of experience in data engineering, with 3+ years hands-on in Databricks.
• Strong proficiency in PySpark, SQL, and Apache Spark.
• Experience with cloud platforms (Azure or AWS) and big data technologies (e.g., Kafka, Hadoop).
• Solid understanding of data warehousing, data modeling, and ETL processes.
• Familiarity with data governance, security, and compliance frameworks.

Preferred Skills:

• Experience with Delta Live Tables (DLT) and Unity Catalog.
• Knowledge of CI/CD pipelines, Git, and DevOps practices.
• Certifications in Databricks, AWS, or Azure are an added advantage.

Soft Skills:

• Excellent communication and collaboration skills.
• Strong problem-solving and analytical mindset.
• Ability to work independently and lead initiatives.

Job Types: Full-time, Permanent

Work Location: In person