Senior Data Pipeline Engineer

Job Title: Senior Data Pipeline Engineer

Location: Remote | Chennai, India

Duration: 6+ months (extendable based on performance)

Experience Required: 8+ years

Work Timing: Flexible (supporting US & India)

Job Overview

We are seeking an experienced Senior Data Pipeline Engineer with strong expertise in modern data platforms. The ideal candidate will have hands-on experience in building, optimising, and scaling large-scale batch and real-time data pipelines using cutting-edge technologies. This role involves working with cross-functional teams to deliver reliable, high-quality data solutions for enterprise-grade applications.

Key Responsibilities

- Design, develop, and maintain scalable and reliable data pipelines for both batch and real-time workloads.

- Implement real-time event streaming and data ingestion using Apache Kafka.

- Build and optimise ETL/ELT workflows on Databricks and integrate with Snowflake for analytics and reporting.

- Write efficient, production-ready code in Python, Node.js, and Java.

- Collaborate with data scientists, analysts, and business stakeholders to deliver consistent data solutions.

- Ensure data quality, governance, and security across platforms.

- Monitor, debug, and optimise distributed data systems for high performance.

- Lead and mentor junior engineers, promoting best practices in data engineering.

Required Qualifications

- 8+ years of professional experience in data engineering or software engineering.

- Strong expertise in PySpark for distributed data processing.

- Hands-on experience with Apache Kafka for data streaming.

- Proficiency with Databricks for big data and ML-driven workloads.

- Experience with Snowflake for cloud data warehousing.

- Advanced programming skills in Python, with proficiency in Node.js and Java.

- Solid knowledge of ETL/ELT frameworks, data modelling, and pipelines.

- Strong understanding of distributed systems and cloud-based architectures.

- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.

Preferred Skills

- Experience with cloud platforms (AWS, Azure, GCP).

- Knowledge of CI/CD pipelines and containerization (Docker, Kubernetes).

- Familiarity with MLOps and data science workflows.

- Strong system design and debugging skills in large-scale environments.

- Excellent communication and problem-solving abilities.

What We Offer

- Opportunity to design and scale enterprise-grade data infrastructure.

- Competitive compensation with career growth opportunities.

- Flexible and collaborative work environment.

Job Type: Contractual / Temporary
Contract length: 6 months

Work Location: In person