Senior Data Engineer

Roles & Responsibilities

• Data Engineering Excellence: Design and implement data pipelines using formats like JSON, Parquet, CSV, and ORC, utilizing batch and streaming ingestion.
• Cloud Data Migration Leadership: Lead cloud migration projects, developing scalable Spark pipelines.
• Medallion Architecture: Implement Bronze, Silver, and gold tables for scalable data systems.
• Spark Code Optimization: Optimize Spark code to ensure efficient cloud migration.
• Data Modeling: Develop and maintain data models with strong governance practices.
• Data Cataloging & Quality: Implement cataloging strategies with Unity Catalog to maintain high-quality data.
• Delta Live Table Leadership: Lead the design and implementation of Delta Live Tables (DLT) pipelines for secure, tamper-resistant data management.
• Customer Collaboration: Collaborate with clients to optimize cloud migrations and ensure best practices in design and governance.

Educational Qualifications

• Experience: Minimum 5 years of hands-on experience in data engineering, with a proven track record in complex pipeline development and cloud-based data migration projects.
• Education: Bachelor’s or higher degree in Computer Science, Data Engineering, or a related field.

Skills

• Must-have: Proficiency in Spark, SQL, Python, and other relevant data processing technologies.
• Strong knowledge of Databricks and its components, including Delta Live Table (DLT) pipeline implementations.
• Expertise in on-premises to cloud Spark code optimization and Medallion Architecture.

Good to Have

• Familiarity with AWS services (experience with additional cloud platforms like GCP or Azure is a plus).

Soft Skills

• Excellent communication and collaboration skills, with the ability to work effectively with clients and internal teams.
• Certifications
• AWS/GCP/Azure Data Engineer Certification.

Job Types: Full-time, Permanent

Work Location: In person