Databricks Developer

Responsibilities

Databricks & Data Engineering

• Design, develop, and maintain end-to-end data pipelines using Databricks (Apache Spark).
• Develop and optimize Spark jobs using PySpark, Scala, or Spark SQL.
• Build and manage Delta Lake tables with ACID transactions, schema enforcement, and time travel.
• Implement batch and streaming data processing using Spark Structured Streaming.
• Optimize performance using partitioning, caching, Z-ordering, cluster tuning, and cost optimization strategies.

Data Ingestion & Integration

• Ingest data from multiple sources such as databases, APIs, cloud storage (ADLS, S3, GCS), and messaging systems.
• Integrate Databricks with ETL/ELT tools, orchestration frameworks, and enterprise systems.
• Work with real-time data sources such as Kafka or Event Hubs (preferred).

Cloud & Platform Management

• Deploy Databricks solutions on AWS, Azure, or GCP.
• Configure and manage Databricks clusters, jobs, workflows, and notebooks.
• Implement security and access controls using IAM, Unity Catalog, and role-based permissions.
• Monitor workloads, troubleshoot failures, and optimize resource utilization.

Analytics & Advanced Use Cases

• Support data analytics and BI use cases by creating curated datasets and optimized views.
• Collaborate with data scientists to support machine learning pipelines, feature engineering, and model training.
• Implement data quality checks, validation frameworks, and monitoring.

Collaboration & Governance

• Work closely with data architects, analysts, data scientists, and business stakeholders.
• Translate business requirements into scalable and maintainable technical solutions.
• Ensure compliance with data governance, privacy, and security standards.
• Participate in code reviews, unit testing, and CI/CD deployments.

Key Skills

• Strong hands-on experience with Databricks and Apache Spark
• Proficiency in PySpark and Spark SQL (Scala is a plus)
• Solid understanding of Delta Lake and Lakehouse architecture
• Experience with data modeling for analytics and reporting
• Strong knowledge of SQL and query optimization techniques
• Experience working on cloud platforms: AWS, Azure, or GCP
• Hands-on experience with:
• Cloud storage: S3, ADLS Gen2, GCS
• Messaging/streaming: Kafka, Event Hubs
• Familiarity with data orchestration tools (Airflow, Azure Data Factory, Databricks Workflows)