Databricks DevOps Lead

Mandatory skills: Databricks, Spark, IAAS, Azure, Python, Cosmos DB, Azure DevOps (GitHub, CI/CD pipelines, Boards etc.), Docker & Azure Kubernetes Service, Graffana, JUnit, Postman, SonarQube
Additional Skills: 5+ years of DevOps/SRE experience supporting data platforms or distributed systems.
Strong experience with CI/CD pipelines (Azure DevOps/GitHub Actions) for Databricks, Spark jobs, and Azure data components. Hands‑on experience with infrastructure as code (Terraform/Bicep/ARM) for provisioning Azure resources (Databricks, ADLS Gen2, Event Hubs, Key Vault). Solid understanding of build and release strategies for notebooks, JARs, wheels, and configuration artifacts. Experience setting up monitoring, logging, and alerting Spark jobs and data pipelines using Azure Monitor/Log Analytics or similar tools.
Strong scripting skills (Python, PowerShell, or Bash) for automation and tooling.
Good to have skills: Experience with containerization and orchestration (Docker, Kubernetes, AKS) for supporting ancillary services. Knowledge of security best practices (managed identities, secrets management, RBAC) for Azure data services. Exposure to reliability engineering concepts (error budgets, SLOs, SLI dashboards) for data workloads.

Roles & Responsibilities:

Ability to create/maintain CI/CD pipelines (Azure DevOps/GitHub Actions) for Databricks, Spark jobs, and Azure data components.
Develop infrastructure as code (Terraform/Bicep/ARM) for provisioning Azure resources (Databricks, ADLS Gen2, Event Hubs, Key Vault).
Create scripts to manage release strategies for notebooks, JARs, wheels, and configuration artifacts.
Set up monitoring, logging, and alerting Spark jobs and data pipelines using Azure Monitor/Log Analytics or similar tools
Coordinate with offshore team in India

TCS Employee Benefits Summary:

Discretionary Annual Incentive.
Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
Family Support: Maternal & Parental Leaves.
Insurance Options: Auto & Home Insurance, Identity Theft Protection.
Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
Time Off: Vacation, Time Off, Sick Leave & Holidays.
Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

#LI-RJ2

Salary Range-$100,000-$130,000 a year

Location

Chicago, IL

Job Function

TECHNOLOGY

Role

Technical Lead

Job Id

398071

Desired Skills

DevOps

Salary Range

$100,000-$130,000 a year

Similar jobs

No similar jobs found