Senior Azure Data Engineer

Job Description

We are seeking an experienced Azure Data Engineer with strong expertise in Azure Data Factory (ADF), Azure Databricks, and preferably Azure Data Fabric to lead the migration and optimization of our data infrastructure. The role involves designing, building, and managing scalable, secure, and efficient data pipelines to support both analytics and GenAI-driven applications.

The ideal candidate will have hands-on experience with large-scale data ingestion pipelines, handling unstructured data, and working on cloud data migrations. If you have successfully executed a migration to Azure Data Fabric or have at least 6 months of hands-on experience with it, you are the perfect fit for this role.

Key Responsibilities

• Assess the current Azure Data Factory and Databricks-based architecture and design a detailed migration plan to Azure Data Fabric.
• Build and optimize end-to-end data pipelines for ingestion, transformation, storage, and analytics.
• Implement multi-model data ingestion pipelines, supporting structured, semi-structured, and unstructured data at scale.
• Ensure scalable ingestion pipelines capable of handling 10,000+ files in production environments.
• Integrate data from SharePoint and other enterprise sources into Azure Data Fabric while maintaining quality, governance, and compliance.
• Work closely with business analysts and stakeholders to support AI/ML and GenAI workloads.
• Implement and maintain data security, access controls, and governance policies across Azure environments.
• Optimize performance, scalability, and cost efficiency of data storage and processing workflows.
• Provide technical leadership, mentoring, and best practices for junior engineers.

Key Requirements

• 5+ years of proven experience in Azure Data Engineering.
• Strong expertise in:
• Azure Data Factory
• Azure Databricks (Spark, PySpark, SQL)
• At least 6 months hands-on with Azure Data Fabric OR successful completion of one migration project to Fabric.
• Experience designing and maintaining data lakes, warehouses, and pipelines in Azure environments.
• Expertise in data transformation, Spark jobs, and SQL optimization.
• Hands-on experience working with large-scale unstructured data ingestion pipelines.
• Experience with OCR-based data extraction workflows is a plus.
• Strong knowledge of data governance, security, compliance, and performance tuning.
• Ability to collaborate with cross-functional teams and communicate effectively with both technical and non-technical stakeholders.
• Preferred Certifications:
• Microsoft Certified: Azure Data Engineer Associate
• Microsoft Certified: Azure Solutions Architect Expert

Job Type: Full-time

Work Location: Remote