Lead II - Software Engineering-Devops Engineer-Databricks

7 - 9 Years
2 Openings
Bangalore, Chennai, Hyderabad, Kochi, Pune, Trivandrum

Role: DevOps Engineer – Databricks Specialist

Role Overview

We are seeking a hands-on DevOps Engineer with deep expertise in Databricks platform administration and a strong background in infrastructure automation, CI/CD, and containerized deployments. The ideal candidate will have a solid foundation in scripting and cloud platform operations, with a focus on managing scalable data and analytics environments.

Key Responsibilities

• Administer and manage the Databricks platform, including:

  • User/workspace management

  • Cluster policies

  • Job scheduling

  • Library management

  • Security controls

• Implement Infrastructure as Code (IaC) using Terraform to provision, configure, and manage cloud resources.

• Build and maintain CI/CD pipelines integrated with Git-based workflows for Databricks and associated services.

• Manage and optimize containerized workloads using Docker and AWS ECS.

• Develop automation scripts and utilities in Python to enhance operational efficiency and streamline data workflows.

• Manage dependency and environment configurations using Poetry.

• Collaborate with data engineers, architects, and platform teams to ensure high availability, scalability, and security.

• Implement monitoring, ing, and logging solutions for proactive platform health management.

• Contribute to DevOps, security, and cost optimization best practices across cloud and data environments.

Required Skills & Experience

• Strong hands-on experience with Databricks platform administration (clusters, jobs, libraries, workspace, security, monitoring).

• Proficiency in Terraform for infrastructure provisioning and automation.

• Solid knowledge of AWS services, including:

  • ECS

  • EC2

  • S3

  • IAM

  • Lambda

  • CloudWatch

• Advanced scripting skills in Python.

• Experience with Docker containerization and orchestration.

• Proficient in Git for version control and team collaboration.

• Experience managing environments and dependencies using Poetry.

• Strong understanding of CI/CD concepts and DevOps best practices.

Terraform,Git,Python,Aws