Lead Data Engineer

Job Designation: Senior Data Ops Engineer – Team Lead

Job Location: Bengaluru

Key Responsibilities:

• Leadership & Team Management
• Lead and mentor a team of DataOps engineers in designing and maintaining robust data pipelines.
• Plan, assign, and review team tasks to ensure timely and quality delivery.
• Collaborate with data engineers, data scientists, and business teams to prioritize data needs and ensure alignment with organizational goals.
• Drive best practices in coding standards, documentation, and deployment automation.
• Technical Delivery
• Design and implement scalable ETL/ELT pipelines using Pentaho, StreamSets, and Python-based frameworks.
• Manage real-time and batch data ingestion using Kafka for streaming and MySQL/Snowflake for storage and transformation.
• Implement and maintain data quality checks, validation, and reconciliation frameworks.
• Ensure pipeline observability, error handling, and alerting mechanisms for proactive issue resolution.
• Optimize Snowflake and MySQL queries for performance and cost efficiency.
• Lead migration or modernization initiatives (e.g., on-prem to Snowflake/cloud).
• Governance & Operations
• Maintain data security, access control, and compliance with enterprise standards.
• Define and track DataOps KPIs such as pipeline success rates, latency, and data quality metrics.
• Partner with Infrastructure and DevOps teams for seamless environment management and scalability.

Technical Skills Required:

Databases:

• Strong expertise in MySQL (query optimization, stored procedures, schema design).
• Advanced knowledge of Snowflake (data modelling, performance tuning, cost optimization).

ETL & Data Pipeline Tools:

• Hands-on experience with Pentaho Data Integration (Kettle) and/or StreamSets for ETL/ELT automation.

Streaming:

• In-depth understanding of Apache Kafka (topic configuration, producer/consumer setup, schema registry, stream processing).

Programming:

• Proficient in Python for data automation, transformation scripts, and integration with APIs.

Monitoring & Observability:

• Familiarity with Grafana, Prometheus, or similar tools for performance and error tracking.

Cloud:

• Exposure to AWS/Azure/GCP data stack (S3, Lambda, Glue, Dataflow, etc.).

Job Type: Full-time

Pay: ₹1,900,000.00 - ₹2,000,000.00 per year

Benefits:

• Commuter assistance
• Health insurance
• Provident Fund

Work Location: In person