Pyspark Data Engineer

We are seeking a Senior Data Engineer with strong expertise in PySpark, Snowflake, and Microsoft Azure to build and optimize scalable, cloud-based data solutions.

Responsibilities

• Design, develop, and optimize data pipelines using PySpark
• Build and manage data ingestion and transformation processes in Azure
• Develop and maintain data models and transformations in Snowflake
• Optimize Snowflake performance (clustering, warehouses, cost optimization)
• Work with large-scale structured and semi-structured datasets
• Implement data quality, validation, and monitoring frameworks
• Collaborate with data architects, analysts, and stakeholders
• Ensure security, governance, and compliance in Azure data solutions
• Troubleshoot and resolve pipeline failures and performance issues
• Mentor junior engineers and contribute to best engineering practices

Requirements:

• 5+ years of experience in Data Engineering
• Strong hands-on experience with PySpark / Apache Spark
• Advanced proficiency in Python
• Strong experience with Snowflake (SQL, performance tuning, cost control)
• Hands-on experience with Azure services, including:
• Azure Data Factory (ADF)
• Azure Data Lake Storage (ADLS Gen2)
• Azure Synapse Analytics
• Strong SQL skills and data modeling knowledge
• Experience with ETL/ELT architectures