Area(s) of responsibility

JD for Azure Data Bricks:

Core responsibilities

Design and development:

Architect and build scalable, robust data solutions on the Azure Databricks platform.

Data pipelines:

Develop and maintain data pipelines, handling both batch and streaming data for various file formats like CSV, JSON, and Parquet.

ETL/ELT processes:

Implement and optimize ETL/ELT processes to transform raw data into usable formats for analytics and reporting.

Collaboration:

Work closely with data engineers, data scientists, and business analysts to meet data requirements and support machine learning workflows.

Performance optimization:

Tune Databricks clusters and jobs for performance, scalability, and cost efficiency.

Integration:

Integrate Databricks with other Azure services such as Azure Data Lake, Synapse, Key Vault, and SQL Database.

Maintenance and support:

Monitor and troubleshoot data pipelines, resolve issues, and perform code reviews.

Security and governance:

Implement security best practices, access controls, and data governance policies.

Required skills and experience

Technical proficiency:

• Strong expertise in Azure Databricks and its components.
• Proficiency in programming languages like PySpark and Python.
• Strong SQL skills.

Azure services: Experience with other Azure services like Azure Data Factory, Azure Data Lake Storage (ADLS), and Azure SQL Database